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B-279992 

April  30, 1999 

The  Honorable  Slade  Gorton 
Chairman 

The  Honorable  John  D.  Rockefeller  IV 
Ranking  Minority  Member 
Subcommittee  on  Aviation 

Committee  on  Commerce,  Science,  and  Transportation 
United  States  Senate 

The  Honorable  John  J.  Duncan 
Chairman 

The  Honorable  William  O.  Lipinski 

Ranking  Democratic  Member 

Subcommittee  on  Aviation 

Committee  on  Transportation  and  Infrastructure 

House  of  Representatives 

In  response  to  your  request,  this  report  addresses  the  extent  to  which  the  Federal  Aviation 
Administration’s  (faa)  Acquisition  Management  System  provides  a  comprehensive  approach  for 
managing  the  agency’s  investments  in  air  traffic  control  information  technology,  faa  plans  to 
spend  billions  of  dollars  to  replace  data-processing,  navigation,  communications,  and  other 
systems  under  its  air  traffic  control  modernization  program  but  has  a  history  of  poor 
performance  in  delivering  systems  on  time  and  within  budget  and  performance  parameters.  We 
found  that  faa  has  established  a  structured  approach  for  managing  its  modernization 
investments,  but  weaknesses  in  this  approach  limit  its  effectiveness.  We  are  making 
recommendations  to  the  Secretary  of  Transportation  to  strengthen  faa’s  investment 
management  approach. 

We  are  sending  copies  of  this  report  to  Senator  Frank  R.  Lautenberg,  Senator  Richard  C.  Shelby, 
Representative  James  A.  Barcia,  Representative  Stephen  Horn,  Representative  Jim  Turner, 
Representative  Constance  A  Morelia,  Representative  Martin  Olav  Sabo,  and  Representative 
Frank  R.  Wolf  in  their  capacities  as  Chair  or  Ranking  Minority  Member  of  Senate  and  House 
Subcommittees.  We  are  also  sending  copies  of  this  report  to  the  Honorable  Rodney  E.  Slater, 
Secretary  of  Transportation;  the  Honorable  Jane  F.  Garvey,  Administrator  of  the  Federal 
Aviation  Administration;  and  the  Honorable  Jacob  Lew,  Director,  Office  of  Management  and 
Budget.  Copies  will  also  be  made  available  to  others  on  request. 


B-279992 


Please  call  me  at  (202)  512-2834  if  you  have  questions  about  the  report.  Major  contributors  to 
this  report  are  listed  in  appendix  II. 


Gerald  L.  Dillingham 
Associate  Director,  Transportation 
Issues 
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Purpose 


The  Federal  Aviation  Administration  (faa)  has  undertaken  an  ambitious 
and  costly  program  to  modernize  its  air  traffic  control  system.  Under  this 
program,  faa  is  acquiring  new  surveillance,  data-processing,  navigation, 
and  communications  equipment  in  addition  to  new  facilities  and  support 
equipment.  Totaling  126  active  projects,  the  modernization  effort  is 
estimated  to  cost  $26.5  billion  from  fiscal  year  1982  through  fiscal  2004.1 
Of  this  total,  faa  estimates  that  it  will  need  $12.9  billion  for  59  information 
technology  projects — the  software-intensive  and  complex  information  and 
communications  systems  supporting  the  air  traffic  control  system. 

Given  the  large  expenditures  required  to  carry  out  faa’s  modernization 
effort,  the  past  problems,  and  the  continuing  concerns  about  key  projects 
funded  under  the  program,  the  Chairmen  and  Ranking  Members  of  the 
Senate  Committee  on  Commerce,  Science,  and  Transportation’s 
Subcommittee  on  Aviation  and  the  House  Committee  on  Transportation 
and  Infrastructure’s  Subcommittee  on  Aviation  asked  gao  to  review  faa’s 
investment  management  approach  as  carried  out  through  the  Acquisition 
Management  System  (ams),  which  was  implemented  in  April  1996.  gao 
evaluated  the  processes,  data,  and  decisions  that  faa  uses  to  select, 
control,  and  evaluate  its  investments.  This  report  addresses  the  extent  to 
which  faa,  through  ams,  (1)  has  established  a  structured  approach  for 
selecting  and  controlling  its  investments;  (2)  incorporates  all  investments, 
including  those  currently  in  operation,  in  the  agency’s  portfolio;  and 
(3)  selects,  controls,  and  evaluates  its  investments  with  complete  and 
reliable  information. 


Results  in  Brief 


faa’s  Acquisition  Management  System  is  a  good  first  step  in  establishing  a 
structured  investment  management  approach  for  selecting  and  controlling 
the  agency’s  investments.  The  system  contains  a  set  of  policies, 
procedures,  and  reporting  requirements  to  analyze  mission  needs;  assess 
the  affordability  of  proposed  projects;  and  establish  life-cycle  costs, 
schedules,  benefits,  and  performance  baselines  (boundaries)  to  control 
the  performance  of  the  projects  that  are  selected.  Additionally,  under  this 
system,  a  senior  management  investment  review  group  makes  key 
decisions  about  which  investments  best  meet  the  agency’s  needs  and  are 
to  be  funded. 

However,  the  system  is  not  comprehensive  in  that  it  does  not  incorporate 
all  of  faa’s  projects  into  a  complete  strategic  investment  portfolio.  Key 


LThe  total  cost  of  the  modernization  program — which  includes  completed,  canceled,  and  restructured 
projects  as  well  as  the  active  projects — is  estimated  to  be  $41  billion  from  fiscal  year  1982  through 
fiscal  2004.  In  this  report,  all  dollars  are  expressed  as  current-year  dollars. 
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decision-making  processes  and  requirements  of  the  Acquisition 
Management  System  are  applied  only  to  proposed  projects  and  those 
under  development  but  not  to  projects  already  in  operation.  In  particular, 
agency  officials  have  not  yet  developed  a  sound  estimate  of  the  costs  to 
operate  projects  and  these  costs  are  not  included  in  the  agency’s  financial 
plan  for  modernization.  Because  faa  does  not  apply  the  same  scrutiny  to 
all  of  its  projects,  senior  officials  are  unable  to  fully  assess  and  make 
trade-offs  about  the  relative  merits  of  spending  funds  to  develop  new 
systems,  to  enhance  current  systems,  or  to  continue  operating  and 
maintaining  existing  systems.  This  report  makes  a  recommendation 
designed  to  strengthen  faa’s  investment  management  by  directing  the 
agency  to  establish  and  control  a  complete  portfolio  of  all  information 
technology  investments,  including  those  projects  already  in  operation. 

faa’s  Acquisition  Management  System  currently  does  not  provide 
complete  and  reliable  information  for  selecting,  controlling,  and  evaluating 
the  agency’s  investments.  First,  the  cost  data  used  to  select  projects  are  of 
questionable  reliability  because  of  weaknesses  in  faa’s  cost  estimating 
practices  and  processes  and  the  lack  of  a  cost  accounting  system.  Second, 
the  information  used  to  control  projects  is  incomplete  since  faa  has  not 
fully  implemented  an  effective  process  for  controlling  the  baselines  for  the 
costs,  schedules,  benefits,  performance,  and  risks  of  its  investments,  faa 
has  approved  the  baseline  information  for  only  half  of  the  required 
universe  of  projects,  and  the  agency’s  processes  for  tracking  actual 
performance  against  estimates  frequently  has  provided  incomplete 
information.  Third,  faa  lacks  information  needed  to  evaluate  its 
investments  since  the  Acquisition  Management  System  does  not  have  a 
post-implementation  evaluation  process  for  assessing  projects’  outcomes 
and  feeding  lessons  learned  back  into  the  selection  and  control  phases  to 
help  improve  its  management  of  future  projects.  This  report  makes  several 
recommendations  to  improve  faa’s  selection,  control,  and  evaluation  of  its 
information  technology  investments. 


Background 


Over  the  past  17  years,  faa’s  modernization  projects  have  experienced 
substantial  cost  overruns,  lengthy  delays,  and  significant  performance 
shortfalls.  Because  of  faa’s  contention  that  some  of  its  modernization 
problems  were  caused  by  federal  acquisition  regulations,  the  Congress 
enacted  legislation  in  November  1995  that  exempted  the  agency  from  most 
federal  procurement  laws  and  regulations  and  directed  faa  to  develop  a 
new  acquisition  management  system.  In  response,  faa  implemented  ams  on 
April  1, 1996.  ams  provides  high-level  acquisition  policy  and  guidance  for 
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selecting  and  controlling  investments  throughout  all  phases  of  the 
acquisition  life  cycle. 

Funding  for  faa’s  modernization  investments  is  primarily  provided  through 
two  of  its  budget  accounts:  (1)  facilities  and  equipment  and  (2)  operations. 
The  facilities  and  equipment  account  covers  the  costs  to  develop,  procure, 
and  place  the  new  equipment  or  facility  in  operation.  Once  the  project 
goes  into  full  operation,  it  is  funded  by  the  operations  account,  which 
covers  the  costs  to  support  and  maintain  the  new  equipment  or  facility. 

Using  the  methodology  described  in  Assessing  Risks  and  Returns:  A  Guide 
for  Evaluating  Federal  Agencies’  IT  Investment  Decision-Making,2  gao 
evaluated  how  faa  selects,  controls,  and  evaluates  its  investments.  This 
guide  incorporates  gao’s  analysis  of  the  management  practices  of  leading 
private  and  public  sector  organizations  as  well  as  provisions  of  major 
federal  legislation  (e.g.,  the  Clinger-Cohen  Act  of  1996)  and  executive 
branch  guidance  that  address  investment  decision-making.  As  part  of  its 
evaluation,  gao  examined  five  of  faa’s  projects  to  determine  how  ams  is 
implemented  at  the  project  level.  (The  five  projects  are  described  in  app. 

I.) 


Principal  Findings 


FAA’s  AMS  Is  Designed  to 
Provide  a  Disciplined, 
Structured  Process  for 
Selecting  and  Controlling 
Investments 


Through  ams,  faa  has  designed  and  implemented  processes  that  provide 
many  of  the  key  elements  leading  organizations  follow  to  select  and 
control  investments  the  agency  funds  through  its  facilities  and  equipment 
budget  account.  In  the  selection  phase,  leading  organizations  take  a 
structured  approach  to  determining  priorities,  screening  and  analyzing  the 
relative  merits  of  the  projects,  and  making  decisions  about  which  projects 
will  be  funded  during  the  year.  Through  ams,  faa  has  established  two 
processes — mission  analysis  and  investment  analysis — that  together 
constitute  a  set  of  policies,  procedures,  and  guidance  that  enhances  the 
agency’s  ability  to  screen  projects  submitted  for  funding;  assess  and  rank 
each  project  based  on  its  relative  costs,  benefits,  risks,  and  contribution  to 
faa’s  mission;  and  utilize  a  senior,  corporate-level  decision-making  group 
to  select  projects  for  funding.  Once  a  project  is  selected,  ams  requires  faa 
officials  to  formally  establish  the  life-cycle  cost,  schedule,  benefits,  and 
performance  baselines  that  are  used  to  monitor  the  project’s  status,  faa 


2GAO/AIMD-10. 1. 13  (Feb.  1997). 
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has  developed  a  number  of  mechanisms  for  monitoring  projects’  estimated 
versus  actual  baseline  performance  and  reporting  any  variances  from  the 
established  baselines. 


Lack  of  Oversight  of  the 
Operations  Portion  of 
Projects  Prevents  FAA 
From  Managing 
Investments  as  a  Complete 
Portfolio 


faa  lacks  oversight  of  the  operations  portion  of  its  investments  under  ams. 
For  example,  faa’s  process  for  scoring  and  ranking  projects  prior  to 
selection  is  applied  to  proposed  projects  and  those  under  development 
that  will  receive  facilities  and  equipment  funding,  not  to  existing  systems 
that  are  funded  from  the  operations  budget  account.  In  contrast,  leading 
organizations  include  all  types  of  information  technology  projects  (i.e., 
new  and  existing  systems)  in  their  selection  process  to  create  a  complete 
strategic  investment  portfolio,  faa  also  has  not  yet  developed  a  sound 
estimate  of  the  operations  cost  baseline  for  each  of  its  projects  and  the 
agency’s  financial  plan  for  the  modernization  program  reports  only  costs 
funded  by  the  facilities  and  equipment  budget  account,  omitting  the 
operations  costs  associated  with  its  investments.  Although  faa  has 
developed  operations  cost  projections  for  26  of  the  70  projects  or 
segments  of  projects  identified  as  requiring  baselines,  officials  throughout 
the  agency  told  us  that  these  estimates  are  not  reliable.  Finally,  while  faa’s 
budget  provides  detailed  analyses  of  actual  and  projected  costs  for  each  of 
the  projects  funded  by  the  facilities  and  equipment  budget  account,  it 
provides  very  little  project-level  detail  in  its  justification  for  the  operations 
budget  account,  faa  has  two  initiatives  that  it  believes  will  improve  the 
data  on  operations  costs.  First,  it  is  developing  a  cost  accounting  system, 
although  operations  data  from  that  system  will  not  be  available  for  at  least 
a  year,  given  the  schedule  for  implementing  the  system.  Second,  faa  has  a 
team  that  is  addressing  the  agency’s  concerns  about  the  quality  of 
operations  data  and  developing  operations  cost  baselines  for  selected 
projects.  This  team  is  expected  to  report  its  findings  in  May  1999. 


Weaknesses  in  the 
Selection,  Control,  and 
Evaluation  Phases  Limit 
FAA’s  Effectiveness  in 
Managing  Its  Investments 


ams  has  weaknesses  in  all  three  investment  management 
phases — selection,  control,  and  evaluation — that  limit  faa’s  ability  to 
manage  its  investments  effectively.  First,  the  cost  information  used  to 
make  selection  decisions  is  of  questionable  reliability,  and  there  is  little 
evidence  that  the  data  or  underlying  analyses  used  in  the  selection  process 
are  validated  to  ensure  accuracy  and  completeness.  While  the  agency  has 
improvements  under  way,  faa’s  cost  estimating  techniques  do  not  yet 
satisfy  recognized  standards  that  call  for  organizing  and  retaining  projects’ 
cost  information  in  a  historical  database  and  using  cost  models  that  are 
calibrated  and  validated  on  the  basis  of  actual  experience.  Instead,  faa’s 
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processes  allow  each  project  to  approach  cost  estimating  in  whatever 
manner  its  estimators  choose.  Moreover,  the  data  used  to  support  faa’s 
selection  decisions  are  not  validated  because  faa  does  not  require  that  all 
project  information — such  as  that  pertaining  to  costs,  schedules, 
performance,  benefits,  or  risks  used  in  making  selection  decisions — be 
validated  under  ams.  Furthermore,  ams  guidance  does  not  specify  what 
types  of  validation  steps  should  be  taken  nor  does  it  require 
documentation  of  the  results. 

Second,  faa  has  not  fully  implemented  its  procedures  for  controlling  key 
projects’  baselines.  To  control  its  projects  at  the  agencywide  level,  faa 
relies  on  periodic  reviews  of  each  project’s  acquisition  program  baseline,  a 
document  that  establishes  a  project’s  cost,  schedule,  benefits,  and 
performance  boundaries  and  that  is  intended  to  be  used  to  monitor  a 
project’s  status  in  achieving  those  boundaries.  This  baseline  document  is 
incomplete,  however,  because  its  schedule  baseline  does  not  include  any 
milestones  for  project  reviews  during  the  operations  phase  of  the  project 
and  because  it  does  not  address  the  project’s  risks.  In  addition,  faa  has 
completed  about  half  of  the  baselines  for  its  universe  of  projects  or  project 
segments  that  require  them,  and  agency-level  processes  for  tracking  actual 
baseline  performance  against  estimates  frequently  provided  incomplete 
information  on  projects’  costs,  schedules,  benefits,  and  performance.  For 
example,  for  the  five  projects  gao  reviewed,  none  of  the  monthly  baseline 
status  reports  analyzed  the  projects’  estimated  operations  costs,  assessed 
estimated  versus  actual  benefits,  or  contained  information  on  the 
performance  requirements  outlined  in  the  projects’  baseline 
documentation.  Moreover,  faa’s  investment  control  group  made  up  of 
senior  managers,  the  Joint  Resources  Council,  is  not  actively  involved  in 
monitoring  all  projects  after  the  investment  decisions  are  made. 

Third,  faa  does  not  have  a  defined,  documented  process  for  conducting 
post-implementation  reviews  of  projects  for  the  purpose  of  assessing 
project  performance  as  well  as  improving  the  selection  and  control  of  its 
investments,  faa  performs  some  elements  of  a  post-implementation  review 
in  its  life-cycle  management  review  process,  including  such  tasks  as  the 
independent  operational  test  and  evaluation  of  some  projects  prior  to 
deployment,  operational  performance  monitoring,  customer  satisfaction 
surveys,  and  periodic  reviews  throughout  an  investment’s  life  cycle. 
However,  this  process  is  not  standardized  and  is  not  required  for  all 
projects.  As  a  result,  there  is  no  evidence  that  changes,  especially  to  the 
selection  and  control  phases,  are  being  implemented  based  on  lessons 
learned. 
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Finally,  faa’s  recently  implemented  agencywide  management  information 
system  for  tracking  information  about  projects  under  ams  contains  data 
related  to  faa’s  processes  for  managing  baselines  but  excludes  key 
selection  data,  such  as  mission  need  statements,  cost-benefit  analyses,  risk 
assessments,  and  other  required  reports.  Informed  management  decisions 
can  only  be  made  if  information  from  all  phases  of  the  investment 
management  process  is  included  in  the  decision-making  process  and  made 
easily  available  through  a  management  information  system. 


Recommendations 


gao  recommends  that  the  Secretary  of  Transportation  direct  the 
Administrator  of  faa  to  implement  a  comprehensive  investment 
management  approach  through  the  Acquisition  Management  System. 
Specifically,  the  Administrator  should  take  the  following  actions: 

Establish  a  complete  portfolio  of  investments — including  existing  systems 
funded  by  the  operations  budget  account  as  well  as  projects  funded  by  the 
facilities  and  equipment  account — and  require  the  Joint  Resources  Council 
to  periodically  review  the  baseline  status  and  merits  of  each  of  these 
investments  throughout  their  entire  life  cycle.  As  part  of  this  portfolio, 
cost  baselines  for  operating  and  maintaining  all  projects  should  be 
developed,  and  this  information  should  be  included  in  the  agency’s 
financial  plan  for  its  investments  and  in  its  annual  budget  request  to  the 
Congress. 

Improve  the  selection  process  by  (1)  establishing  clearly  defined 
procedures  for  validating  each  project’s  cost,  schedule,  benefit, 
performance,  and  risk  information  and  (2)  requiring  documentation  of  the 
results  of  the  validation  procedures  applied  to  each  project. 

Strengthen  control  over  investments  by  (1)  revising  the  acquisition 
program  baseline  requirements  to  include  project  risks  and  to  add 
milestones  for  project  reviews  during  the  operations  phase  and 
(2)  ensuring  that  project  officials  fully  track  and  document  estimated 
versus  actual  results  on  all  the  elements  (i.e.,  cost,  schedule,  benefit, 
performance,  and  risk)  contained  in  the  baseline  documentation. 

Initiate  post-implementation  evaluations  for  projects  within  3  to  12  months 
of  deployment  or  cancellation  to  compare  the  completed  projects’  cost, 
schedule,  performance,  and  mission  improvement  outcomes  with  the 
original  estimates. 

Incorporate  key  information  from  the  selection  process  (e.g.,  mission  need 
statements,  cost-benefit  analyses,  and  risk  assessments)  into  faa’s 
management  information  system  for  investments. 
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Agency  Comments 
and  Our  Evaluation 


We  provided  copies  of  a  draft  of  this  report  to  the  Department  of 
Transportation  and  faa  for  their  review  and  comment.  We  met  with  faa 
officials,  including  the  Associate  Administrator  for  Research  and 
Acquisitions,  who  is  also  faa’s  Acquisition  Executive.  These  officials 
generally  agreed  with  the  recommendations  in  this  report  and  made 
clarifying  comments,  which  have  been  incorporated  as  appropriate,  faa 
officials  noted  that  the  Acquisition  Management  System  represents  a 
substantial  revision  to  the  way  that  faa  contracts  for  large  and  complex 
systems  and  that  it  is  important  to  maintain  perspective  on  how  far  the 
agency  has  come  in  a  relatively  short  time.  These  officials  were  concerned 
that  the  use  of  gao’s  guide,  Assessing  Risks  and  Returns:  A  Guide  for 
Evaluating  Federal  Agencies’  IT  Investment  Decision-Making,  unduly 
diminished  faa’s  accomplishments  by  comparing  the  agency  to  an  ideal 
end-state  that  may  not  exist  in  any  single  organization.  They  also  asserted 
that  the  Acquisition  Management  System  would  compare  favorably  with 
acquisition  systems  in  other  federal  government  or  private  sector 
organizations. 

As  we  stated  in  this  report,  faa’s  Acquisition  Management  System  is  a  good 
first  step  in  establishing  a  structured  investment  management  approach, 
but  it  has  weaknesses  that  limit  its  effectiveness.  The  gao  guide  was 
developed  to  provide  a  structure  for  evaluating  and  assessing  how  well  a 
federal  agency  is  managing  its  information  technology  resources  and  to 
identify  specific  areas  where  improvements  can  be  made.  The  concepts 
and  practices  contained  in  the  guide  are  based  on  the  practices  followed 
by  leading  private  and  public  sector  organizations  as  well  as  on  provisions 
of  major  federal  legislation  and  executive  branch  guidance  that  address 
investment  decision-making.  While  acknowledging  the  wide  variance 
among  organizations  and  the  complexity  of  the  investment  management 
process,  the  guide  focuses  on  the  common  elements  that  should  be 
present  in  any  organization’s  investment  management  process.  Our  review 
evaluated  the  extent  to  which  faa’s  Acquisition  Management  System 
contains  these  elements,  not  how  well  faa’s  system  compares  with  other 
federal  acquisition  systems.  Therefore,  we  did  not  make  changes  to  the 
report  based  on  these  comments. 


Page  10 


GAO/RCED/AIMD-99-88  FAA’s  Investment  Management 


Page  11 


GAO/RCED/AIMD-99-88  FAA’s  Investment  Management 


Contents 


Executive  Summary 


4 


Chapter  1 
Introduction 


14 

The  ATC  System  Modernization  Program.  Is  Complex,  Costly,  and  15 
Historically  Problematic 

FAA  Has  Developed  the  Acquisition  Management  System  to  16 

Manage  Its  Modernization  Investments 
FAA's  Acquisitions  Are  Funded  From  Two  Major  Budget  18 

Accounts 

Guidance  Provides  a  Framework  for  Assessing  Federal  Agencies’  19 
Information  Technology  Investment.  Decision-Making 
Objectives,  Scope,  and  Methodology  20 


Chapter  2 
FANs  AMS  Is 
Designed  to  Provide  a 
Disciplined, 
Structured  Process 
for  Selecting  and 
Controlling 
Investments 


FAA  Has  Established  a  Defined  Process  for  Selecting  Investments 
Funded  Through  the  Facilities  and  Equipment  Account 
AMS  Requires  That  Projects’  Baselines  Be  Established  and 
Controlled 
Conclusions 


22 

22 

24 

26 


Chapter  3 

Lack  of  Oversight  of 
the  Operations 
Portion  of  Projects 
Prevents  FAA  From 
Managing  Investments 
as  a  Complete 
Portfolio 


FAA’s  Selection  Process  Does  Not  Address  Operations-Funded 
Investment  Projects 

FAA  Does  Not  Have  a  Complete  and  Sound  Operations  Cost 
Baseline  for  Its  Investments 

FAA’s  Operations  Budget  Justifies  Only  a  Small  Portion  of  Its 
Spending  for  Investments 
Conclusions 
Recommendation 
Agency  Comm  ents 


27 

27 

28 

29 

30 
30 
30 


Page  12 


GAO/RCED/AIMD-99-88  FAA’s  Investment  Management 


Contents 


Chapter  4 
Weaknesses  in  the 
Selection,  Control, 
and  Evaluation 
Phases  Limit  FAA’s 
Effectiveness  in 
Managing  Its 
Investments 


Weaknesses  in  Some  Supporting  Data  Limit  the  Effectiveness  of 
the  Selection  Process 

FAA  Has  Not  Fully  Implemented  Its  Process  for  Establishing  and 
Tracking  Key  Project  Baselines 
FAA’s  AMS  Lacks  a  Post-Implementation  Review  Process  for 
Evaluating  Investments 

FAA’s  Efforts  to  Implement  an  Agencywide  Management 
Information  System  Do  N  ot  Include  Key  Selection  Data 
Conclusions 
Recommendations 
Agency  Comments 


31 

31 

33 

41 

42 

44 

44 

45 


Appendixes 


Appendix  1:  Background  and  Status  of  Five  Projects 
Appendix  II:  Major  Contributors  to  This  Report 


46 

53 


Tables  Table  2.1:  Information  on  the  Baseline  Elements  Required  in  the  25 

Acquisition  Program  Baseline  and  the  Project  Parameter  Sheet 
Table  4, 1:  Status  of  FAA’s  Efforts  to  Develop  AMS  Baseline  37 

Documentation  on  60  Ongoing  Projects  and  Segments  for  Which 
Investment  Decisions  Have  Been  Made 
Table  4.2:  Analysis  of  the  Missing  Baseline  Information.  Reported  39 

in  Acquisition  Reviews  of  Five  Projects 


Figure 


Figure  1.1:  FAA’s  Life-Cycle  Acquisition  Management  Process 


18 


Abbreviations 

ams  Acquisition  Management  System 

atc  air  traffic  control 

faa  Federal  Aviation  Administration 

gao  General  Accounting  Office 

i-beam  Integrated  Baseline  Establishment  and  Management 

nas  National  Airspace  System 

spire  Simplified  Program  Information  Reporting  and  Evaluation 


Page  13 


GAO/RCED/AIMD-99-88  FAA’s  Investment  Management 


Chapter  1 


Introduction 


The  Federal  Aviation  Administration’s  (faa)  mission  is  to  promote  the 
safe,  orderly,  and  expeditious  flow  of  air  traffic  in  the  United  States 
through  what  is  commonly  referred  to  as  the  National  Airspace  System 
(nas).  faa’s  ability  to  fulfill  its  mission  depends  on  the  adequacy  and 
reliability  of  the  nation’s  air  traffic  control  (atc)  system — the  principal 
component  of  the  nas — which  comprises  a  vast  network  of  radars; 
automated  data-processing,  navigation,  and  communications  equipment; 
and  atc  facilities.  It  is  through  the  atc  system  that  faa  provides  services 
such  as  controlling  takeoffs  and  landings  and  managing  the  flow  of  traffic 
between  airports.3  Sustained  growth  in  air  traffic  and  faa’s  aging 
equipment  have  strained  the  current  atc  system.  This  growth  in  traffic  is 
predicted  to  continue  as  the  number  of  passengers  traveling  on  U.S. 
airlines  is  expected  to  grow  from  about  674  million  in  1998  to  1.065  billion 
by  2010,  an  increase  of  about  57  percent. 

To  relieve  the  problems  of  aging  equipment  and  to  accommodate  the 
predicted  growth  in  air  traffic,  faa  initiated  a  multibillion-dollar 
modernization  effort  in  December  1981.  Our  work  over  the  years  has 
chronicled  many  of  faa’s  failures  in  meeting  projects’  cost,  schedule,  and 
performance  goals.  Because  of  the  size,  complexity,  cost,  and 
problem-plagued  past  of  faa’s  modernization  program,  we  have  designated 
it  a  high-risk  information  technology  investment  since  1995.4 


3FAA  uses  three  types  of  facilities  to  manage  and  control  traffic.  Airport  towers  direct  aircraft  on  the 
ground,  before  landing,  and  after  takeoff  when  they  are  about  5  nautical  miles  from  the  airport  and  up 
to  about  3,000  feet  above  the  airport.  Terminal  radar  approach  control  facilities  sequence  and  separate 
aircraft  as  they  approach  and  leave  busy  aiiports,  beginning  about  5  nautical  miles  and  ending  about  50 
nautical  miles  from  the  aiiport  and  generally  up  to  10,000  feet  above  the  ground.  Air  route  traffic 
control  centers,  called  en  route  centers,  control  planes  in  transit  and  during  approaches  to  some 
airports.  Most  of  the  en  route  centers’  controlled  airspace  extends  above  18,000  feet  for  commercial 
aircraft.  En  route  centers  also  handle  aircraft  at  lower  altitudes  when  dealing  directly  with  a  tower  or 
when  agreed  upon  with  a  terminal  facility.  FAA  provides  additional  services,  such  as  weather  and  pilot 
briefings,  through  a  network  of  flight  service  stations. 

4FAA’s  modernization  program  is  one  of  four  high-risk  system  development  and  modernization  efforts 
in  the  federal  government.  See  High-Risk  Series:  An  Overview  (GAO/HR-95-1,  Feb.  1995),  High-Risk 
Series:  Information  Management  and  Technology  (GAO/HR-97-9,  Feb.  1997),  and  High-Risk  Series:  An 
Update  (GAO/HR-99-1,  Jan.  1999). 
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The  ATC  System 
Modernization 
Program  Is  Complex, 
Costly,  and 
Historically 
Problematic 


Under  its  ambitious  modernization  program,  faa  is  acquiring  new 
surveillance,  data-processing,  navigation,  and  communications  equipment 
in  addition  to  new  facilities  and  support  equipment.  Totaling  126  active 
projects,  the  modernization  is  estimated  to  cost  $26.6  billion  from  fiscal 
year  1982  through  fiscal  2004.6  Of  this  total,  faa  estimates  that  it  will  need 
$12.9  billion  from  fiscal  year  1982  through  2004  for  59  information 
technology  projects — the  software-intensive  and  complex  information  and 
communications  systems  supporting  the  atc  system.  These  projects  range 
from  those  designed  to  replace  equipment  used  by  controllers  to 
communicate  with  aircraft  and  with  each  other  to  radars  that  provide 
controllers  with  surveillance  information  for  separating  aircraft.  An 
example  of  an  information  technology  project  is  the  Display  System 
Replacement  project  that  will  modernize  equipment  in  faa’s  en  route 
facilities  by  replacing  20-  to  30-year-old  display  channels,  controllers’ 
workstations,  and  network  infrastructure. 


Over  the  past  17  years,  faa’s  modernization  projects  have  experienced 
substantial  cost  overruns,  lengthy  delays,  and  significant  performance 
shortfalls.  To  illustrate,  the  longtime  centerpiece  of  the  modernization 
program — the  Advanced  Automation  System — was  restructured  in  1994 
after  estimated  costs  to  develop  the  system  tripled  from  $2.5  billion  to  $7.6 
billion  and  delays  in  putting  significantly  less-than-promised  system 
capabilities  into  operation  were  expected  to  run  8  years  or  more  over  the 
original  estimates.6  These  problems  have  persisted.  For  example,  two  key 
projects  in  the  modernization  effort — the  Wide  Area  Augmentation  System 
and  the  Standard  Terminal  Automation  Replacement  System — have 
encountered  significant  cost  increases,  delays,  and  changes  in 
requirements.7  In  the  case  of  the  Wide  Area  Augmentation  System, 
between  September  1997  and  January  1998,  total  estimated  costs 
increased  by  $600  million,  or  25  percent,  from  $2.4  billion  to  $3  billion.  The 
increased  costs  were  attributable  to  faa’s  including  previously  overlooked 
costs  for  periodically  updating  the  project’s  equipment  and  to 
higher-than-expected  operations  and  maintenance  costs.  In  the  case  of  the 
Standard  Terminal  Automation  Replacement  System,  although  faa  has  not 
officially  changed  the  project’s  baseline  approved  in  February  1996,  the 


“The  total  cost  of  the  modernization  program — which  includes  completed,  canceled,  and  restructured 
projects  as  well  as  active  projects — is  estimated  to  be  $41  billion  from  fiscal  year  1982  through  fiscal 
2004.  In  this  report,  all  dollars  are  expressed  as  current  year  dollars. 

6See  Advanced  Automation  System:  Implications  of  Problems  and  Recent  Changes 
(GAO/T-RC  ED-94-1 88 ,  Apr.  13, 1994).  “ 

7See  Air  Traffic  Control:  Observations  on  FAA’s  Modernization  Program  (GAO/T-RCED/AIMD-98-93, 
Feb.  26, 1998). 
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baseline  is  in  jeopardy  of  being  breached  because  of  unions’  concerns 
surrounding  human-factor  and  design  issues,8  the  refinement  of 
requirements,  and  the  interjection  of  a  new  project  phase.9  faa  estimates 
that  these  issues  have  the  potential  to  increase  the  project’s  costs  from 
$294  million  to  $410  million  above  the  approved  baseline,  faa  also 
estimates  that  the  project’s  initial  completion  could  be  delayed  by  almost 
2-1/2  years. 


FAA  Has  Developed 
the  Acquisition 
Management  System 
to  Manage  Its 
Modernization 
Investments 


Because  of  faa’s  contention  that  some  of  its  modernization  problems  were 
caused  by  federal  acquisition  regulations,  the  Congress  enacted  legislation 
in  November  1995  that  exempted  the  agency  from  most  federal 
procurement  laws  and  regulations  and  directed  faa  to  develop  and 
implement  a  new  acquisition  management  system  that  would  address  the 
unique  needs  of  the  agency.10  On  April  1, 1996,  in  response  to  the 
Congress’s  action,  faa  implemented  a  new  acquisition  management 
system.  The  system  is  intended  to  reduce  the  time  and  cost  to  field  new 
products  and  services  by  introducing  (1)  a  new  investment  management 
system  that  spans  the  entire  life  cycle  of  an  acquisition,  (2)  a  new 
procurement  system  that  provides  flexibility  in  selecting  and  managing 
contractors,  and  (3)  organizational  and  cultural  reform  that  supports  the 
new  investment  and  procurement  systems. 

The  Acquisition  Management  System  (ams)  provides  high-level  acquisition 
policy  and  guidance  for  selecting  and  controlling  faa’s  investments 
throughout  all  phases  of  the  acquisition  life  cycle,  which  is  organized  into 
a  series  of  phases  and  decision  points,  including  (1)  mission  analysis, 

(2)  investment  analysis,  (3)  solution  implementation,  (4)  in-service 
management,  and  (5)  service  life  extension,  ams  provides  guidance  on  the 
documents  and  decisions  that  result  from  each  of  these  phases.  For 
example,  through  the  mission  analysis  process,  faa  identifies  critical  needs 
that  the  agency  must  meet  for  improving  the  safety,  security,  capacity, 
efficiency,  and  effectiveness  of  the  nas.  Approval  of  a  mission  need 
statement  by  the  Joint  Resources  Council — faa’s  corporate 
decision-making  body — signifies  that  the  agency  agrees  that  the  need  is 
critical  enough  to  proceed  to  the  next  phase,  investment  analysis.  During 
the  investment  analysis  phase,  teams  of  acquisition  and  program 
specialists  (1)  identify  and  analyze  alternatives,  (2)  develop  baselines  and 


8Concems  were  raised  by  two  unions,  the  National  Air  Traffic  Controllers  Association  and  the 
Professional  Airways  Systems  Specialists. 

9See  Air  Traffic  Control:  Status  of  FAAs  Modernization  Program  (GAO/RCED-99-25,  Dec.  3, 1998). 

l0Department  of  Transportation  and  Related  Agencies  Appropriations  Act  of  1996  (P.L.  104-50). 
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assesses  affordability,  (3)  prepare  an  investment  analysis  report,  and 
(4)  recommend  a  preferred  solution  to  the  mission  need,  faa  then  scores 
and  ranks  each  proposed  project  based  on  a  number  of  factors,  including 
how  well,  relative  to  other  projects,  it  meets  mission  needs  and  has  a 
favorable  cost-benefit  ratio.  Once  a  project  is  selected,  life-cycle  cost, 
schedule,  benefits,  and  performance  baselines  are  established  in  a  formal 
document  called  the  acquisition  program  baseline.  The  acquisition 
program  baseline,  which  must  be  approved  by  the  Joint  Resources 
Council,  is  used  to  monitor  a  project’s  status  in  achieving  those  baselines 
throughout  the  remaining  phases  of  the  acquisition  management  life  cycle. 

During  the  solution  implementation  phase,  a  multidisciplinary  team 
develops  and  carries  out  an  acquisition  strategy  for  implementing  the 
project.  Once  the  project  has  been  implemented  and  is  in  operation  (the 
in-service  management  phase),  the  team  monitors  and  assesses  its 
performance,  costs,  and  support  trends;  proposes  fixes  for  any  defects  or 
other  problems;  incorporates  product  improvements;  seeks  new 
technology  to  enhance  the  capability  or  reduce  costs;  and  identifies  and 
prepares  for  decisions  to  correct  capability  shortfalls  at  the  end  of  the 
project’s  service-life.  Finally,  during  the  service-life  extension  phase,  a 
determination  is  made  about  whether  the  current  capability  satisfies  the 
demand  for  services  or  whether  another  solution  offers  the  potential  for 
improving  safety  or  effectiveness  or  for  significantly  lowering  costs.  The 
team  initiates  a  process  whereby  the  mission  need  would  be  revalidated 
and  the  investment  analysis  process  begun  again,  which  could  lead  to  a 
new  investment  decision.  See  figure  1.1  for  a  graphic  depiction  of  faa’s 
life-cycle  acquisition  management  process. 
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Figure  1.1:  FAA’s  Life-Cycle  Acquisition  Management  Process 
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Guidance  Provides  a 
Framework  for 
Assessing  Federal 
Agencies’  Information 
Technology 
Investment 
Decision-Making 


Several  recent  management  reforms — including  the  revision  of  the 
Paperwork  Reduction  Act  and  the  passage  of  the  Clinger-Cohen  Act  of 
1996,  the  Government  Performance  and  Results  Act  of  1993,  and  the  Chief 
Financial  Officers  Act  of  1990 — have  introduced  requirements 
emphasizing  the  need  for  federal  agencies  to  improve  their  management 
processes  for  selecting  and  managing  information  technology  resources. 
gao  and  the  Office  of  Management  and  Budget  have  developed  guidance  to 
assist  federal  agencies  in  evaluating  information  technology  investments. 
One  such  guide,  Assessing  Risks  and  Returns:  A  Guide  for  Evaluating 
Federal  Agencies’  IT  Investment  Decision-Making,  incorporates  our 
analysis  of  the  management  practices  of  leading  private  and  public  sector 
organizations  as  well  as  of  the  provisions  of  major  federal  legislation  (e.g., 
the  Clinger-Cohen  Act)  and  executive  branch  guidance  that  address 
investment  decision-making.11  The  guide  outlines  three  phases  of  a 
successful  investment  management  approach — selection,  control,  and 
evaluation.  To  help  ensure  that  real,  positive  change  is  produced  as 
agencies  seek  to  improve  their  decision-making  about  their  information 
technology  investments,  agencies  need  to  (1)  institutionalize  management 
processes;  (2)  regularly  validate  the  cost,  benefit,  and  risk  data  used  to 
support  information  technology  decisions;  and  (3)  focus  on  measuring  and 
evaluating  results.  The  guide  provides  a  framework  for  evaluating  and 
assessing  how  well  a  federal  agency  is  achieving  these  goals  and  identifies 
specific  areas  where  improvements  can  be  made. 


Many  of  the  concepts  in  our  Assessing  Risks  and  Returns  have  been 
incorporated  into  the  Office  of  Management  and  Budget’s  Capital 
Programming  Guide,  which  provides  guidance  on  the  planning,  budgeting, 
acquisition,  and  management  of  different  kinds  of  capital  assets,  including 
information  technology.12  Our  guide  has  also  been  endorsed  by  the  federal 
Chief  Information  Officers  Council.  A  third  guide  we  prepared,  Executive 
Guide:  Leading  Practices  in  Capital  Decision-Making,  summarizes  12 
fundamental  practices  that  have  been  successfully  implemented  by 
organizations  recognized  for  their  outstanding  capital  decision-making.13 
Since  information  technology  investments  are  a  form  of  capital  asset,  this 
guide  emphasizes  many  of  the  same  concepts  as  the  aforementioned 
guides,  such  as  evaluating  alternative  approaches  to  achieving  results; 


UGAO/AIMD-10. 1. 13  (Feb.  1997). 

i2CapitaL  Programming  Guide,  Version  1.0,  Supplement  to  Office  of  Management  and  Budget  Circular 
A-Il,  Executive  Office  of  the  President,  Office  of  Management  and  Budget  (July  1997). 

i3Executive  Guide:  Leading  Practices  in  Capital  Decision-Making  (GAO/AIMD-99-32,  Dec.  1998). 
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assessing  investments  as  a  portfolio;  tracking  projects’  costs,  schedules, 
and  performance;  and  conducting  post-implementation  reviews. 


Objectives,  Scope, 
and  Methodology 


Given  the  large  expenditures  required  to  carry  out  faa’s  modernization 
program,  the  past  problems,  and  the  continuing  concerns  about  key 
projects  funded  under  the  program,  the  Chairmen  and  Ranking  Minority 
Members  of  the  Senate  Commerce,  Science,  and  Transportation 
Committee’s  Subcommittee  on  Aviation  and  the  House  Transportation  and 
Infrastructure  Committee’s  Subcommittee  on  Aviation  asked  us  to 
evaluate  the  extent  to  which  ams  provides  a  comprehensive  approach  for 
managing  faa’s  modernization  investments.  This  report  addresses  the 
extent  to  which  faa,  through  ams,  (1)  has  established  a  structured 
approach  for  selecting  and  controlling  its  investments;  (2)  incorporates  all 
investments,  including  those  in  operation,  in  the  agency’s  portfolio;  and 
(3)  selects,  controls,  and  evaluates  its  investments  with  complete  and 
reliable  information. 

To  address  our  objectives,  we  reviewed  faa’s  overall  approach  for 
managing  investments,  as  carried  out  through  ams,  including  the  policies, 
procedures,  and  guidance  for  managing  the  life-cycle  acquisition  process. 
ams  is  designed  to  include  all  of  faa’s  acquisition  projects,  including  those 
related  to  facilities,  mission  support,  and  information  technology.  Our 
review  focused  primarily  on  information  technology  projects,  and  we 
worked  with  faa  to  identify  the  universe  of  such  projects.  Then,  in  concert 
with  faa,  we  selected  five  projects  for  review  to  obtain  a  more  detailed 
understanding  of  how  the  investment  management  process  is  implemented 
at  the  project  level.  The  five  projects  we  reviewed  were  (1)  the  nas 
Infrastructure  Management  System,  (2)  the  Oceanic  Automation  Program, 
(3)  the  Operational  and  Supportability  Implementation  System,  (4)  the 
Standard  Terminal  Automation  Replacement  System,  and  (5)  the  Voice 
Switching  and  Control  System.  We  selected  these  projects  because  they 
were  in  various  stages  of  implementation  when  ams  was  established  in 
April  1996,  were  key  to  replacing  the  aging  nas  infrastructure  or  improving 
its  capacity  and  effectiveness,  and  represented  significant  expenditures. 
Total  estimated  facilities  and  equipment  funding  for  each  of  these  projects 
exceeds  $100  million.  Two  of  the  five  projects — the  nas  Infrastructure 
Management  System  and  the  Operational  and  Supportability 
Implementation  System — are  subject  to  all  ams  requirements  because  they 
were  selected  after  ams  went  into  effect  in  April  1996.  The  other  three 
projects,  which  were  ongoing  when  ams  was  established,  are  subject  to  ams 
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requirements  for  the  remaining  stages  in  their  development  and 
implementation. 

As  part  of  our  effort,  we  applied  the  methodology  prescribed  in  Assessing 
Risks  and  Returns.  We  used  the  questions  in  this  guide  to  determine  the 
extent  to  which  faa  has  decision-making  and  management  processes  and 
data  in  place  to  select  information  technology  projects  and  systems, 
control  these  projects  throughout  their  life  cycles,  and  evaluate  results  and 
revise  the  processes  based  on  lessons  learned.  We  provided  the  questions 
in  the  guide  to  faa  officials  and  assessed  their  responses,  along  with 
supporting  documentation,  as  a  basis  for  determining  whether  faa’s 
approach  provided  the  necessary  elements  for  managing  its  investments. 
We  also  incorporated  the  guidance  from  the  Executive  Guide:  Leading 
Practices  in  Capital  Decision-Making  in  determining  whether  faa  managed 
its  investments  as  a  portfolio;  tracked  projects’  costs,  schedules,  and 
performance;  and  conducted  post-implementation  reviews. 

To  gain  an  overall  perspective  on  faa’s  investment  management  process, 
we  interviewed  faa  officials  responsible  for  implementing  and  managing 
ams.  We  also  interviewed  faa  officials  responsible  for  preparing  the 
facilities  and  equipment  and  operations  budgets  and  reviewed  the  agency’s 
fiscal  year  1999  budget  justification  documentation  to  understand  how 
data  on  faa’s  investments  are  used  to  support  faa’s  budget  request  to  the 
Congress.  Finally,  we  interviewed  officials  responsible  for  managing  the 
five  projects  we  reviewed  to  obtain  their  views  on  how  ams  is  applied  at 
the  project  level  as  well  as  potential  areas  for  improvement.  We  performed 
our  work  from  April  1998  through  April  1999  in  accordance  with  generally 
accepted  government  auditing  standards. 
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Through  ams,  faa  has  designed  and  implemented  processes  that  provide 
many  of  the  key  elements  followed  by  leading  organizations  for  selecting 
and  controlling  investments,  ams’  mission  analysis  and  investment  analysis 
processes  are  meant  to  provide  faa’s  senior  management  with  a  basis  for 
screening  proposed  projects;  evaluating  their  relative  costs,  benefits,  and 
risks;  and  selecting  projects  for  funding  based  on  their  relative  merits. 
Additionally,  ams  policy  requires  each  acquisition  project  to  have  an 
approved  baseline  that  establishes  the  project’s  life-cycle  cost,  schedule, 
benefits,  and  performance  boundaries  and  that  is  intended  to  be  used  to 
monitor  the  project’s  status  in  achieving  those  baselines. 


FAA  Has  Established 
a  Defined  Process  for 
Selecting  Investments 
Funded  Through  the 
Facilities  and 
Equipment  Account 


During  the  selection  phase,  leading  organizations  take  a  structured 
approach  to  determining  priorities,  screening  and  analyzing  the  relative 
merits  of  projects,  and  making  decisions  about  which  projects  will  be 
funded  during  the  year.  Such  an  approach  builds  on  an  organization’s 
assessment  of  where  it  should  invest  its  resources  for  the  greatest  benefit 
over  the  long  term.  The  Clinger-Cohen  Act  requires  federal  agencies  to 
apply  this  sort  of  structured  approach  in  deciding  whether  to  undertake 
particular  investments  in  information  technology  systems. 

A  starting  point  for  the  selection  phase  is  the  screening  process  in  which 
projects  being  submitted  for  funding  are  compared  against  a  uniform  set 
of  screening  criteria  and  thresholds  to  determine  whether  the  projects 
meet  minimal  requirements  and  to  identify  at  what  organizational  level  the 
projects  should  be  reviewed.  Next,  the  costs,  benefits,  risks,  and  mission 
focus  of  all  the  projects  are  assessed,  and  the  projects  are  compared 
against  each  other  and  ranked  or  prioritized.  In  conducting  their  selection 
processes,  leading  organizations  require  all  projects  to  have  complete  and 
accurate  project  proposals  and  justification  information.  Finally,  a 
decision-making  body  of  senior  managers  makes  decisions  about  which 
projects  to  select  for  funding  on  the  basis  of  mission  needs  and 
organizational  priorities.  The  selection  phase  helps  ensure  that  the 
organization  selects  those  projects  that  will  best  support  mission  needs 
and  identifies  and  analyzes  each  project’s  risks  and  proposed  benefits 
before  a  significant  amount  of  funds  is  spent. 


Through  ams,  faa  has  established  two  processes — mission  analysis  and 
investment  analysis — that  together  constitute  a  set  of  policies,  procedures, 
and  guidance  that  are  designed  to  enhance  the  agency’s  ability  to  select 
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investments.14  These  two  processes  define  what  should  be  done  and  who 
should  do  it,  what  reports  are  required,  who  reviews  and  approves  reports 
and  processes,  who  makes  corporate-level  decisions,  and  the  roles  and 
responsibilities  of  those  involved,  ams  policy  and  guidance — which  is  on 
an  easily  accessible  Internet  Website — contain  the  procedures,  process 
flowcharts,  document  templates,  checklists,  and  other  acquisition-related 
information  needed  for  faa’s  project  officials  and  senior  management  to 
understand  and  implement  the  selection  processes. 

Although  ams  does  not  include  an  explicit  screening  step,  screening 
activities  are  part  of  the  agency’s  mission  analysis  process.  This  process 
culminates  in  a  mission  need  statement  reflecting  the  Joint  Resources 
Council’s  decision  that  a  high-priority,  critical  need  exists  and  that  the 
agency  should  go  forward  with  a  detailed  investment  analysis  of  proposed 
solutions  to  meet  that  need.  During  mission  analysis,  faa’s  operating 
divisions  identify  and  quantify  projected  demand  for  and  supply  of 
services,  capability  shortfalls,  and  technological  opportunities  to  meet 
those  shortfalls,  and  summarize  the  major  decision  factors  that  the  Joint 
Resources  Council  should  evaluate  in  considering  the  need.  The  Joint 
Resources  Council  approves  mission  need  statements. 

Once  a  mission  need  statement  is  approved,  ams’  investment  analysis 
process  provides  a  set  of  detailed  steps  for  evaluating  the  costs,  benefits, 
and  risks  of  alternative  solutions  and  for  selecting  the  best  solution  to 
meet  the  need.  Under  ams,  multidisciplinary  investment  analysis 
teams — comprising  officials  from  the  operating  divisions  and  other 
acquisition  and  engineering  specialists — assess  each  project  proposed  for 
funding  to  define  the  technical  requirements;  estimate  the  life-cycle  costs, 
benefits,  schedule,  and  risks;  and  determine  the  project’s  affordability 
relative  to  other  projects.  As  part  of  the  investment  analysis  process,  faa 
scores  and  ranks  each  proposed  project  on  the  basis  of  defined  criteria  for 
how  well — relative  to  other  projects — it  meets  the  agency’s  mission 
objectives,  whether  it  is  of  high  priority  to  the  organization  sponsoring  the 
project,  whether  it  is  consistent  with  the  nas  architecture15  or  provides 
critical  administrative  capacity  for  the  agency,  and  whether  it  has  a 
favorable  cost-benefit  ratio,  ams  requires  that  the  mission  need  be 
revalidated  during  the  investment  analysis  and  that  the  underlying 
analyses  be  documented  through  cost-benefit  studies,  risk  assessments, 
and  other  documents. 


i4As  we  discuss  in  chapter  3,  key  aspects  of  AMS’  selection  process  are  limited  to  projects  funded  by 
the  facilities  and  equipment  account  and  exclude  investments  funded  by  the  operations  account. 

i5The  architecture  is  FAA’s  blueprint  for  defining  the  long-range  needs  of  the  NAS. 
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After  considering  the  results  of  the  investment  analysis  process,  the  Joint 
Resources  Council  decides  whether  to  proceed  with  a  proposed  project. 
This  step  is  called  the  investment  decision,  and  if  the  project  is  selected, 
the  Council  commits  faa  to  fully  funding  it.  The  project  is  then 
incorporated  into  the  agency’s  financial  plan  for  projects  funded  by  the 
facilities  and  equipment  budget  account. 


AMS  Requires  That 
Projects’  Baselines  Be 
Established  and 
Controlled 


Under  ams,  faa’s  primary  mechanism  for  controlling  a  project  is  the 
acquisition  program  baseline,  which  establishes  a  project’s  life-cycle  cost, 
schedule,  benefits,  and  performance  baselines  and  which  is  intended  to  be 
used  to  monitor  a  project’s  status  in  achieving  those  baselines.  The 
acquisition  program  baseline  is  supposed  to  be  established  when  the 
investment  decision  is  made  for  a  project.  The  baseline,  which  is  the 
agreement  between  the  organizations  within  faa  that  are  acquiring  and 
will  use  the  project,  sets  the  cost  and  schedule  boundaries  within  which 
the  project  is  authorized  to  proceed,  defines  the  performance  and  benefits 
the  project  must  achieve,  and  establishes  the  performance  measurements 
for  assessing  the  project’s  success  as  it  advances  through  its  life  cycle. 


faa’s  ultimate  goal  is  to  establish  an  acquisition  program  baseline 
document  for  every  acquisition  project.  When  ams  was  established  in 
April  1996,  however,  the  agency  decided  to  establish  two  standards  for 
baseline  documentation  to  facilitate  the  preparation  of  baselines  during 
faa’s  transition  to  ams.  One  standard  applies  to  those  projects  initiated 
after  ams  and  another,  less  detailed  standard  applies  to  those  projects 
ongoing  at  the  time  that  ams  was  established.  The  post-AMs  projects  must 
have  an  acquisition  program  baseline  that  provides  a  full  set  of  detailed 
information  on  each  baseline  element  as  well  as  a  document  called  a 
parameter  sheet  that  summarizes  a  subset  of  baseline  information  that  is 
designated  for  the  Joint  Resources  Council’s  control  and  that  is 
considered  critical  to  assessing  the  project’s  ability  to  satisfy  mission 
need,  achieve  needed  operational  capability,  achieve  benefits,  and  meet 
the  schedule  requirements  of  interdependent  programs.  Projects  begun 
before  ams  was  established  are  required  only  to  have  the  parameter  sheet. 
Table  2.1  summarizes  the  full  set  of  baseline  information  contained  in  the 
acquisition  program  baseline  and  the  subset  of  information  contained  in 
the  parameter  sheet. 
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Table  2.1:  Information  on  the  Baseline  Elements  Required  in  the  Acquisition  Program  Baseline  and  the  Project  Parameter 

Sheet 

Baseline 

elements 

Full  set  of  baseline  information  contained  in  the  acquisition 
program  baseline 

Subset  of  baseline  information  contained  in 
the  parameter  sheet 

Cost 

Life-cycle  costs  are  broken  down  by  (1)  all  sources  of 
appropriations  funding,  (2)  all  fiscal  years  over  the  project's 
entire  life  cycle,  and  (3)  12  detailed  cost  elements  (program 
management,  testing  and  evaluation,  training,  data  management, 
physical  integration,  systems  and  equipment,  implementation, 
product  support,  operations  and  maintenance,  in-service 
support,  in-service  monitoring  and  assessment,  and  disposal  of 
replaced  assets). 

Total  life-cycle  costs  are  broken  down  by  source 
of  appropriations  funding. 

Schedule 

Schedule  includes  a  list  of  all  events  related  to  satisfying  mission 
need,  providing  intended  operational  capability,  and  accruing 
benefits  as  well  as  events  crucial  to  other  related  NAS  programs 
or  systems. 

Schedule  includes  a  subset  of  events  most 
crucial  to  satisfying  mission  need,  providing 
operational  services,  and  accruing  benefits— for 
example,  contract  award,  in-service  decision 
(the  point  at  which  the  new  system  is  certified 
ready  for  operational  use)— and  the  dates  that 
the  first  and  last  sites  are  commissioned  into 
operational  use. 

Benefits 

Total  life-cycle  benefits  are  listed  for  both  the  government  and 
users,  with  specific  measures  for  evaluating  the  annual  economic 
benefits  and  whether  these  benefits  have  been  achieved. 

Benefits  include  total  life-cycle  economic 
benefits. 

Performance 

Performance  includes  all  requirements  for  project  milestones; 
technical  performance;  acquisition  of  technical  systems, 
equipment,  facilities,  and  services;  coordination  with  outside 
organizations;  and  operations  support  and  management 
requirements. 

Performance  includes  the  total  number  of 
systems,  sites,  or  services  provided. 

Additionally,  it  includes  other  performance 
requirements  deemed  most  critical  to  achieving 
operational  effectiveness,  accruing  benefits,  or 
meeting  other  dependent  NAS  needs  and  those 
considered  to  pose  the  greatest  risk  for  cost  or 
schedule  growth. 

Source:  FAA. 


At  the  agencywide  level,  faa  has  a  two-tiered  process  for  monitoring 
projects’  estimated  versus  actual  baselines.  Under  one  process,  known  as 
the  Integrated  Baseline  Establishment  and  Management  (i-beam)  process, 
the  project  officials  and  officials  responsible  for  overseeing  projects’ 
baselines  monitor  the  projects  and  prepare  reports.  Project  officials 
prepare  a  monthly  status  report  that  is  supposed  to  analyze  estimated 
versus  actual  results  for  the  subset  of  baseline  information  that  is 
monitored  by  the  Joint  Resources  Council  and  summarized  in  the 
parameter  sheets  prepared  for  every  project.  Project  officials  must  also 
inform  the  Joint  Resources  Council  of  any  baseline  violation  or  “breach” 
through  a  document  called  a  baseline  management  notice  and  receive  the 
Council’s  approval  for  any  change  in  the  baselines.  Additionally,  faa’s 
acquisition  oversight  officials  prepare  a  monthly  report  that  analyzes  the 
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status  reports,  baseline  management  notices,  and  other  documents  to 
identify  baseline  variances  and  report  this  information  to  the  Joint 
Resources  Council.  Under  a  second  process,  the  Acquisition  Executive 
who  heads  the  Council  and  other  senior  managers  conduct  acquisition 
reviews  that  are  designed  to  periodically  examine  the  status  of  the 
information  in  each  project’s  acquisition  program  baseline,  along  with 
assessing  a  project’s  progress,  risks,  and  other  issues. 

In  addition  to  these  two  primary  control  processes,  faa  also  has  a  variety 
of  other  reporting  systems  for  monitoring  projects’  status,  including 
executive-level  metrics  that  analyze  project  performance,  project-level 
reviews,  and  numerous  contractor  reports  on  costs,  schedule,  and 
technical  performance.  However,  these  various  reporting  systems  do  not 
analyze  a  project’s  status  in  meeting  all  of  the  detailed  baseline 
requirements  contained  in  the  acquisition  program  baselines  and 
parameter  sheets. 


Conclusions 


faa  has  taken  a  positive  step  in  establishing  an  investment  management 
approach  through  ams.  Under  ams,  faa  has  developed  structured  processes 
for  selecting  and  controlling  its  investments,  ams  provides  a  defined  set  of 
policies,  procedures,  and  reporting  requirements  that  are  designed  to 
facilitate  faa’s  efforts  to  analyze  mission  needs,  identify  alternative 
solutions  to  meet  those  needs,  assess  the  solutions’  affordability,  and 
establish  and  control  a  project’s  performance  once  a  solution  is  selected. 
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faa’s  oversight  of  its  investments  is  confined  to  new  projects  and  those 
under  development,  limiting  its  ability  to  fully  assess  and  make  trade-offs 
between  new  and  existing  systems  and  preventing  the  agency  from 
managing  all  of  its  investments  as  a  complete  portfolio,  faa’s  oversight 
includes  the  agency’s  processes  for  scoring  and  ranking  investments  prior 
to  selection  as  well  as  its  processes  for  establishing  and  monitoring 
financial  baselines  after  a  project  has  been  selected.  But  this  oversight  is 
limited  to  projects  that  receive  funding  from  the  facilities  and  equipment 
budget  account — that  is,  new  projects  and  those  under  development — and 
excludes  projects  funded  by  the  operations  account.  Moreover,  the  link 
between  faa’s  investment  management  process  and  its  budget  process  is 
limited  to  the  facilities  and  equipment  budget  account,  which  has  detailed 
information  on  investments  that  does  not  exist  for  the  operations  budget 
account,  faa  has  two  ongoing  initiatives  that  may  improve  its  information 
on  operations  costs. 

Leading  organizations  determine  priorities  and  make  decisions  about 
which  projects  will  be  funded  based  on  analyses  of  the  relative  costs, 
benefits,  and  risks  of  all  the  projects  in  their  investment  portfolios, 
including  projects  that  are  proposed,  under  development,  and  in 
operation.  Such  a  portfolio  approach  allows  the  organizations  to  evaluate 
the  relative  merits  of  spending  funds  to  develop  new  systems,  enhance 
current  systems,  or  continue  operating  and  maintaining  existing  systems. 


FAA’s  Selection 
Process  Does  Not 
Address 

Operations-Funded 
Investment  Projects 


faa’s  process  for  scoring  and  ranking  projects  prior  to  selection  is  applied 
to  proposed  projects  and  those  under  development  that  will  receive 
facilities  and  equipment  funding,  not  to  existing  systems  that  are  funded 
from  the  operations  budget  account.  In  contrast,  leading  organizations 
include  all  types  of  information  technology  projects  in  their  selection 
process  to  create  a  complete  strategic  investment  portfolio.  By  analyzing 
the  entire  portfolio,  the  senior  managers  of  any  organization  can  examine 
the  costs  of  maintaining  existing  systems  versus  investing  in  new  ones; 
comparatively  rank  projects  based  on  expected  costs,  benefits,  and  risks; 
and  reach  decisions  based  on  projects’  overall  contribution  to  the  most 
pressing  organizational  needs,  faa  has  not  included  operations-funded 
projects  in  its  scoring  and  ranking  process  because,  as  discussed  below, 
the  agency  lacks  reliable  data  on  actual  operations  costs  for  existing 
systems  and  projections  of  operations  costs  for  new  projects. 
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While  faa’s  policy  requires  baseline  cost  estimates  for  the  full  life  cycles  of 
all  projects  under  ams,  in  practice,  faa  has  not  yet  developed  a  sound 
estimate  of  the  operations  costs  for  each  of  its  projects.  As  of 
February  1999,  the  agency  had  developed  operations  cost  projections  for 
only  26  of  the  70  projects  identified  by  faa  as  requiring  an  acquisition 
program  baseline  or  parameter  sheet  under  ams.  For  existing  systems 
already  in  the  operations  phase  when  ams  was  implemented  in  April  1996, 
faa  also  lacked  information  on  life-cycle  cost  projections  for  operating  the 
systems  in  the  future  or  actual  operations  costs  incurred  for  each  of  these 
systems.  Moreover,  faa  officials  throughout  the  organization  indicated  that 
the  estimates  of  operations  costs  that  have  been  developed  for  projects 
are  not  reliable.  Given  the  lack  of  data  on  operations  costs,  faa’s  financial 
plan  for  its  modernization  effort  reports  only  costs  funded  by  the  facilities 
and  equipment  budget  account,  omitting  the  operations  costs  associated 
with  its  investments. 

faa  has  long  recognized  the  need  to  project  the  operations  costs  of  its 
projects  over  a  multiyear  period.  In  March  1993,  faa  issued  guidance  under 
its  predecessor  acquisition  system  that  required  an  annual  operations  plan 
to  support  long-range  resource  allocation  planning.  This  plan — which  was 
never  prepared — was  supposed  to  summarize  the  operations  and  support 
funding  requirements  of  modernization  projects  and  other  acquisitions 
funded  by  the  operations  appropriation.  The  operations  plan  was 
supposed  to  contain  a  financial  baseline  that  identified  detailed  operations 
requirements  over  a  5-year  period  and  general  requirements  for  an 
additional  10  years. 

faa  officials  told  us  that  the  agency  lacks  the  information  needed  to 
reliably  estimate  operations  costs  over  a  project’s  life  cycle  or  to  track 
actual  operations  costs  against  estimates  because  it  does  not  have  a  cost 
accounting  system.  In  January  1997,  we  reported  that  faa  lacked  reliable 
cost  estimating  processes  and  cost  accounting  practices  needed  to 
effectively  manage  its  investments  in  information  technology.16  We 
concluded  that,  as  a  result,  the  Congress  does  not  have  reliable  cost 
information  to  use  in  making  decisions  about  faa’s  billion-dollar 
modernization  investments. 

faa  has  two  initiatives  that  it  believes  will  improve  data  on  operations 
costs.  In  August  1998,  faa  formed  an  operations  baseline  team  to  address 
agency  wide  concerns  about  the  quality  of  its  estimates  of  the  operations 


i6See  Air  Traffic  Control:  Improved  Cost  Information  Needed  to  Make  Biilion-Dollar  Modernization 
Investment  Decisions  (GAO/AIMD-97-20,  Jan.  22,  1997). 
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costs  for  modernization  projects  and  the  lack  of  integration  between  the 
facilities  and  equipment  and  operations  budget  accounts.  This  team  is 
evaluating  faa’s  current  processes  for  estimating  and  reporting  on 
operations  costs,  assessing  the  validity  of  operations  cost  data  on  a  sample 
of  projects,  and  exploring  ways  to  improve  the  estimating  process.  The 
team,  which  expects  to  report  its  findings  in  May  1999,  plans  to  develop  a 
10-year  operations  cost  baseline  for  selected  projects  and  to  recommend 
revisions  to  the  budget  formulation  process  that  will  allow  faa  to  budget 
for  the  operations  costs  of  both  new  and  existing  systems  and  to  better 
address  the  interrelationships  between  the  facilities  and  equipment  and 
operations  budget  accounts. 

faa  is  also  developing  a  cost  accounting  system  that  it  believes  will 
provide  more  reliable  information  on  actual  operations  costs;  however, 
faa  has  missed  its  initial  milestones  for  completing  the  new  system’s 
design  and  generating  improved  operations  cost  data  According  to 
officials  responsible  for  the  new  cost  accounting  system,  the  agency  had 
planned  to  begin  accumulating  data  for  domestic  and  oceanic  air  traffic 
services  by  October  1998.  faa  officials  indicated  that  they  underestimated 
the  complexity  of  developing  the  system  and  that  they  now  expect  to 
accumulate  data  for  air  traffic  services  by  April  2000  and  to  fully 
implement  the  system  by  April  2001. 


FAA’s  Operations 
Budget  Justifies  Only 
a  Small  Portion  of  Its 
Spending  for 
Investments 


While  faa’s  budget  provides  detailed  analyses  of  the  actual  and  projected 
costs  for  each  of  the  projects  funded  by  the  facilities  and  equipment 
budget  account,  it  provides  very  little  project-level  detail  in  its  justification 
for  the  operations  budget  account.  In  its  fiscal  year  1999  President’s 
budget  request,  for  example,  faa  justified  only  5  percent,  or  $295  million, 
of  its  total  $5.6  billion  operations  budget  account.  This  $295  million 
request — which  represented  the  incremental  increase  over  the  prior  year’s 
appropriation — contained  information  on  all  of  the  activities  to  be  funded 
in  a  given  year,  including  pay  increases  for  the  current  staff,  costs  of  hiring 
new  staff,  training,  accident  investigations,  and  other  activities. 


Among  the  details  provided  for  the  fiscal  year  1999  operations  budget 
account,  faa  provided  project-  or  system-level  justifications  for  about  1 
percent,  or  $62  million,  of  its  total  $5.6  billion  operations  request.  This 
information,  known  as  the  “nas  Handoff”  portion  of  the  operations 
account,  contains  1-year  estimates  for  new  equipment,  systems,  and 
facilities  that  were  initially  acquired  with  facilities  and  equipment  funding 
but  will  now  be  funded  under  the  operations  budget  account.  The  nas 
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Handoff  contains  details  on  a  variety  of  projected  operations  costs, 
including  controller  overtime,  logistics,  systems  maintenance,  leased 
communications,  and  flight  inspections  and  procedures. 

One  factor  contributing  to  the  different  levels  of  detail  for  the  two  budget 
accounts  is  the  lack  of  reliable  data  on  operations  costs  for  individual 

projects  or  systems.  Also,  faa  officials  told  us  that  they  only  justify 
incremental  increases  over  the  prior  year’s  operations  budget  account 
because  that  is  traditionally  all  that  the  Congress  requires  of  faa  in 
preparing  its  budget  justifications. 

Conclusions 

ams’  lack  of  oversight  of  the  operations  portion  of  faa’s  investments 
impedes  the  agency’s  ability  to  rigorously  assess  and  manage  all  of  its 
modernization  projects  as  a  complete  strategic,  investment  portfolio  and 
to  make  sound  decisions  about  continuing,  modifying,  or  canceling 
projects.  Excluding  operations  projects  from  its  selection  process 
prevents  faa  from  considering  the  relative  merits  of  existing  systems  when 
deciding  which  projects  to  fund  each  year. 

Recommendation 

We  recommend  that  the  Secretary  of  Transportation  direct  the 
Administrator  of  faa  to  establish  a  complete  portfolio  of 
investments — including  existing  systems  funded  by  the  operations  budget 
account  as  well  as  projects  funded  by  the  facilities  and  equipment 
account — and  to  require  the  Joint  Resources  Council  to  periodically 
review  the  baseline  status  and  merits  of  each  of  these  investments 
throughout  their  entire  life  cycles.  As  part  of  this  portfolio,  cost  baselines 
for  operating  and  maintaining  all  projects  should  be  developed,  and  this 
information  should  be  included  in  the  agency’s  financial  plan  for  its 
investments  and  in  its  annual  budget  request  to  the  Congress. 

Agency  Comments 

faa  agreed  with  our  recommendation. 
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Weaknesses  in  the  Selection,  Control,  and 
Evaluation  Phases  Limit  FAA’s  Effectiveness 
in  Managing  Its  Investments 


ams  has  weaknesses  in  all  three  investment  management 
phases — selection,  control,  and  evaluation — that  limit  faa’s  ability  to 
effectively  manage  its  modernization  investments.  First,  the  information 
used  to  select  projects  is  not  validated  to  ensure  quality  control,  and 
critical  cost  information  used  to  support  selection  decisions  is  of 
questionable  reliability.  Second,  faa  has  not  fully  implemented  an  effective 
process  for  controlling  the  baselines  for  the  cost,  schedule,  benefits, 
performance,  and  risks  of  its  investments.  Third,  faa  lacks  a 
post-implementation  evaluation  process  for  assessing  projects’  outcomes 
and  feeding  lessons  learned  back  into  the  selection  and  control  phases  for 
future  projects.  Finally,  faa  has  not  fully  implemented  a  standardized 
management  information  system  for  capturing  and  maintaining  consistent, 
reliable,  and  easily  accessible  data  on  investments. 


Weaknesses  in  Some 
Supporting  Data  Limit 
the  Effectiveness  of 
the  Selection  Process 


ams’  processes  for  selecting  investments  contain  the  key  elements  that 
leading  organizations  follow  to  ensure  the  selection  of  projects  that 
enhance  mission  performance  and  that  are  cost-effective;  however, 
weaknesses  in  some  of  the  data  used  to  support  the  selection  processes 
limit  ams’  effectiveness,  ams’  mission  and  investment  analysis  processes 
provide  faa’s  senior  management  with  a  basis  for  screening  proposed 
projects;  evaluating  their  relative  costs,  benefits,  and  risks;  and  selecting 
projects  for  funding  based  on  their  relative  merits.  But  the  cost 
information  used  to  make  selection  decisions  is  of  questionable  reliability, 
and  there  is  little  evidence  that  the  data  or  underlying  analyses  used  in  the 
selection  process  are  validated  to  ensure  accuracy,  completeness,  and 
appropriateness.  As  a  result,  faa’s  managers  cannot  be  assured  that  they 
have  all  of  the  information  needed  to  make  sound  selection  decisions. 


Cost  Data  Are  of 
Questionable  Reliability, 
Though  Improvements  Are 
Under  Way 


Consistently  producing  reliable  cost  estimates  for  projects  requires 
defining  institutional  processes  for  deriving  estimates  and  measuring 
actual  performance  against  these  estimates.  However,  the  cost  data  used 
in  faa’s  selection  process  are  of  questionable  reliability.  The  five  projects 
we  reviewed  prepared  their  cost  estimates  using  techniques  and  data  that 
we  criticized  in  our  January  1997  report.17  We  found  that  faa’s 
modernization  program’s  cost  estimating  processes  do  not  satisfy 
recognized  standards  and  that  the  agency  does  not  have  a  cost  accounting 
system  capable  of  reliably  accumulating  full  cost  information  for  projects. 
faa’s  cost  estimating  techniques  do  not  satisfy  recognized  standards  that 
call  for  organizing  and  retaining  cost  information  on  projects  in  a 


17GAO/AIMD-97-20  (Jan.  22,  1997). 
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historical  database  and  using  cost  models  that  are  calibrated  and  validated 
on  the  basis  of  actual  experience,  faa’s  processes  allow  each  project  to 
approach  cost  estimating  in  whatever  manner  its  estimators  choose.  The 
result  is  inconsistency  in  the  rigor  and  discipline  with  which  cost 
estimates  are  derived,  which  in  turn  means  estimates  vary  in  their  degrees 
of  reliability.  For  example,  of  the  six  projects  reviewed  in  our  1997  report, 
two  were  too  poorly  documented  to  permit  any  comparative  analysis,  and 
none  of  the  remaining  four  satisfied  all  the  recognized  standards. 
Compounding  the  estimating  weaknesses  is  faa’s  practice  of  presenting 
cost  estimates  as  precise  point  estimates,  thus  failing  to  disclose  the 
estimates’  inherent  uncertainty  and  risks.  Moreover,  the  effectiveness  of 
faa’s  cost  estimating  processes  also  relies  heavily  on  the  quality  of 
projects’  actual  cost  information,  but  faa  does  not  have  a  cost  accounting 
system  for  capturing  and  reporting  the  full  costs  of  its  projects. 
Consequently,  faa  cannot  reliably  use  information  about  actual  cost 
experiences  to  improve  its  future  cost  estimating  efforts.  We 
recommended  that  faa  institutionalize  defined  cost  estimating  processes 
that  include,  among  other  items,  a  historical  database  and  structured 
approaches  and  tools. 

In  response  to  our  1997  report,  faa  is  developing  a  cost  estimating  process 
for  its  projects  that  is  intended  to  satisfy  recognized  estimating  standards; 
drafting  guidance  on  reporting  projects’  cost  estimates  as  ranges  rather 
than  precise  point  estimates  and,  in  fact,  reporting  ranges  on  some 
systems;  and  developing  a  cost  accounting  system.  Additionally,  faa  has 
developed  a  document,  known  as  a  standard  work  breakdown  structure,18 
which  provides  a  good  first  step  toward  the  development  of  a  historical 
database  on  costs,  faa  officials  also  indicated  that  they  are  completing  a 
cost  estimating  handbook  that  contains  a  detailed  discussion  of  cost 
estimating  practices.  When  completed,  this  handbook  should  contribute  to 
improving  faa’s  approach  to  estimating  projects’  costs.  However,  it  does 
not  require  a  disciplined  process  for  estimating  costs  throughout  the 
agency,  and  the  draft  handbook  acknowledges  that  faa  still  needs  to 
develop  sophisticated  tools  and  a  historical  database  to  advance  its  cost 
estimating  processes,  faa  has  not  established  firm  deadlines  for 
completing  the  handbook  or  the  other  tasks  related  to  cost  estimating.  As 
for  the  cost  accounting  system,  as  we  discuss  earlier,  faa  officials 
underestimated  the  complexify  of  developing  the  system  and  found  that 
their  implementation  milestones  were  unrealistic.  The  agency  plans  to 
fully  implement  the  cost  accounting  system  by  April  2001. 


lsThe  work  breakdown  structure,  which  contains  a  detailed  list  of  activities  to  be  accomplished  in 
carrying  out  projects,  is  used  to  develop  life-cycle  cost  estimates  for  projects. 
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Selection  Data  Are  Not  Leading  organizations  validate  the  information  and  analyses  submitted  in  a 

Validated  new  project  proposal,  which  helps  to  ensure  that  all  the  information  is 

up-to-date,  cost  numbers  are  accurate,  benefits  are  quantified  to  the  extent 
possible,  alternatives  are  identified,  underlying  assumptions  are 
reasonable,  and  sensitivity  analyses  are  conducted.  Validation  is  also 
important  for  ensuring  that  a  project’s  risks  are  identified,  that  the  impact 
of  risks  on  the  project’s  outcomes  is  quantitatively  or  qualitatively 
projected,  and  that  risk  mitigation  strategies  are  explained.  Explicit 
verification  and  validation  steps  sensitize  decisionmakers  to  important 
factors  that  have  a  bearing  on  projects’  actual  outcomes,  enhance 
accountability,  and  decrease  the  likelihood  that  project  proposals  will 
contain  analyses  that  are  based  on  inaccurate  or  incomplete  data  or  faulty 
assumptions. 

faa  has  not  completely  defined  the  requirements  for  the  validation  process 
under  ams,  and  it  is  not  fully  carrying  out  validation  activities.  First,  ams 
does  not  require  the  validation  of  all  data  used  in  the  selection  process. 
Under  ams  guidance,  the  faa  organization  charged  with  carrying  out  the 
investment  analysis  phase  is  responsible  for  validating  only  the  cost  and 
schedule  data,  not  the  performance,  benefits,  or  risk  analyses  used  as  part 
of  the  selection  process.  Second,  ams  guidance  does  not  specify  what  steps 
should  be  taken  in  validating  selection  data,  nor  does  it  require 
documentation  of  the  results  and  resolution  of  the  validation  process. 
Finally,  our  review  of  the  two  projects  for  which  investment  decisions  had 
been  made  under  ams  indicated  that  faa  is  not  fully  carrying  out  validation 
activities.  For  one  project,  faa  did  not  provide  any  documentary  evidence 
of  the  validation  efforts  that  were  performed.  For  the  other  project,  faa 
provided  evidence  of  a  validation  review  of  the  cost-benefit  analysis,  but 
most  of  the  results  of  that  review  were  not  incorporated  into  the  final 
version  of  the  analysis  used  to  support  the  investment  decision. 


FAA  Has  Not  Fully 
Implemented  Its 
Process  for 
Establishing  and 
Tracking  Key  Project 
Baselines 


To  control  its  projects  at  the  agencywide  level,  faa  relies  on  periodic 
reviews  of  each  project’s  acquisition  program  baseline,  which,  as  noted 
earlier,  is  a  document  that  establishes  a  project’s  cost,  schedule,  benefits, 
and  performance  boundaries  and  is  intended  to  be  used  to  monitor  a 
project’s  status  in  achieving  those  baselines.  This  document  is  incomplete, 
however,  because  its  schedule  baseline  does  not  include  any  milestones 
for  project  reviews  during  the  operations  phase  of  a  project  and  because  it 
does  not  address  a  project’s  risks.  Moreover,  faa  has  not  completed  about 
half  of  the  baselines  for  the  projects  or  project  segments  that  require 
baselines.  Additionally,  the  agency-level  processes  for  tracking  actual 
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baseline  performance  against  estimates  frequently  provided  incomplete 
information  on  projects’  costs,  schedules,  benefits,  and  performance. 
Finally,  the  investment  control  group  of  senior  managers,  the  Joint 
Resources  Council,  is  not  actively  involved  in  monitoring  all  projects  after 
the  investment  decisions  are  made.  As  a  result,  faa’s  senior  managers  lack 
key  information  needed  to  make  sound  decisions  about  the  future  of  each 
project. 

Leading  organizations  maintain  a  cycle  of  continual  control  and 
monitoring  after  a  project  has  been  selected.  Senior  executives  review  a 
project  at  specific  milestones  as  the  project  moves  through  its  life  cycle 
and  as  the  dollars  spent  on  the  project  increase.  At  these  milestones,  the 
executives  compare  the  expected  costs,  risks,  and  benefits  of  earlier 
phases  with  the  actual  costs  incurred,  risks  encountered,  and  benefits 
realized  to  date.  During  the  control  phase,  senior  executives  determine 
whether  projects  should  be  modified,  continued,  accelerated,  delayed,  or 
terminated.  The  Clinger-Cohen  Act  also  stresses  the  importance  of 
consistently  monitoring  the  progress  of  federal  investments  in  information 
technology  in  meeting  cost,  schedule,  and  performance  objectives. 


FAA’s  Requirements  for 
Baseline  Estimates  Include 
Most,  but  Not  All,  Key 
Project  Parameters 


Our  review  indicated  that  faa’s  acquisition  program  baseline  is  designed  to 
capture  sufficient  information  on  most  of  a  project’s  key  baseline 
elements,  except  for  two  limitations  in  the  areas  of  schedule  and  risk.  One 
limitation  is  that  the  schedule  baseline  does  not  address  the  operations 
phase  of  the  project.  At  leading  organizations,  even  after  a  project  has 
been  implemented,  senior  managers  regularly  review  how  well  the 
acquired  system  meets  organizational  needs,  including  whether  it  needs 
unexpected  modifications  or  premature  replacement  to  meet  emerging 
needs.  These  reviews  are  used  to  make  decisions  pertaining  to  the 
retirement  or  replacement  of  systems.  In  addition,  because  operations 
activities — such  as  hardware  upgrades,  system  software  changes,  ongoing 
training,  and  maintenance  costs — can  consume  a  significant  level  of 
resources,  a  plan  for  the  review  of  each  project  should  be  developed  and 
periodically  reevaluated.  Given  that  ams  is  designed  to  track  projects 
during  their  entire  life  cycles,  we  expected  to  see  one  or  more  milestones 
in  the  schedule  baseline  for  a  project  review  during  the  operations  phase 
of  faa’s  information  technology  investments,  which  can  last  as  long  as  15 
years.  Such  milestones  would  allow  the  Joint  Resources  Council  to  review 
projects  periodically  during  the  operations  phase  to  determine  whether 
expected  performance  requirements  and  benefits  are  actually  being 
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achieved  in  a  cost-effective  manner  and  to  evaluate  the  need  for  upgrading 
or  enhancing  the  technology. 

The  second  limitation  is  that  the  acquisition  program  baseline  does  not 
include  a  project’s  expected  risks.  The  Office  of  Management  and  Budget’s 
guidance  requires  that  federal  agencies  estimate  a  project’s  risks  and 
develop  a  strategy  for  mitigating  those  risks.19  While  faa  performs  a  risk 
analysis  as  apart  of  the  investment  analysis  phase,  the  agency  does  not 
systematically  monitor  each  project’s  risks  during  the  control  phase.  Given 
that  the  acquisition  program  baseline  is  faa’s  primary  mechanism  for 
controlling  projects,  we  expected  it  to  include  a  baseline  assessment  of 
risks  that  could  be  used  to  monitor  mitigation  and  resolution  of  actual 
risks  that  occur  during  a  project’s  life  cycle.  Although  all  five  projects  we 
reviewed  had  discussions  of  risk  issues  during  their  project  reviews,  none 
of  them  presented  any  systematic  assessment  of  estimated  versus  actual 
risks  because  risk  is  not  a  required  element  of  the  acquisition  program 
baseline. 

Requiring  an  assessment  of  a  project’s  risks  as  an  element  of  the 
acquisition  program  baseline  and  systematically  monitoring  those  risks 
would  allow  faa  to  identify  “red  flag”  issues  that  may  have  an  impact  on  a 
project’s  cost,  schedule,  and  performance.  For  example,  two  of  the 
projects  we  reviewed — the  Standard  Terminal  Automation  Replacement 
System  and  the  nas  Infrastructure  Management  System — have  experienced 
difficulties  with  acquiring  or  developing  software,  faa  promoted  both  of 
these  as  projects  that  would  use  commercial  off-the-shelf  software  that 
would  require  very  little  software  development.  For  both  projects,  faa 
underestimated  the  lines  of  software  code  that  needed  to  be  developed  or 
modified,  and  as  a  result,  the  costs  for  the  projects  have  increased  and  the 
schedules  have  been  delayed,  faa  has  historically  had  difficulty  acquiring 
software — the  most  costly  and  complex  component  of  information 
technology  systems — and  the  agency  has  initiated  some  efforts  to  improve 
its  software  acquisition  processes.20  Improvements  in  these  processes, 
coupled  with  an  identification  of  the  risks  associated  with  software 
acquisition  for  new  projects,  would  help  faa  better  manage  such  risks. 


l9See  Office  of  Management  and  Budget  Circular  No.  A-ll,  Part  3,  Planning,  Budgeting,  and  Acquisition 
of  Capital  Assets,  Executive  Office  of  the  President  (June  1997),  and  Capital  Programming  Guide, 
Version  1  07 Supplement  to  Office  of  Management  and  Budget  Circular  A-ll  (July  1997). 

20See  Air  Traffic  Control:  Immature  Software  Acquisition  Processes  Increase  FAA  System  Acquisition 
Risks  (GAO/AIMD-97-47,  Mar.  21,  1997). 
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FAA  Has  Approved  Half  of 
Its  Required  Baselines,  and 
Some  Baseline 
Documentation  Is 
Incomplete 


faa  has  identified  70  modernization  projects  or  project  segments  that 
require  baseline  documentation.21  These  70  projects  and  segments  account 
for  $1,266  billion,  or  55  percent,  of  the  $2,319  billion  appropriation  faa 
requested  for  fiscal  year  2000  for  all  modernization  activities  funded  by  the 
facilities  and  equipment  budget  account.  Most  of  the  modernization 
activities  that  do  not  have  baselines  involve  improving  or  sustaining  atc 
facilities  and  other  buildings;  providing  technical  support  services;  or 
funding  personnel,  compensation,  benefits,  and  travel  costs. 


Of  the  70  projects  and  segments,  10  involve  projects  that  are  currently  in 
the  investment  analysis  phase  of  ams  and,  hence,  do  not  yet  require  an 
approved  baseline.  Thus,  60  projects  and  segments  that  account  for  $1,205 
billion  in  fiscal  year  2000  estimated  costs,  currently  require  a  baseline.  Of 
the  60,  half  have  approved  baseline  documentation  (either  acquisition 
program  baselines  or  parameter  sheets).22  These  30  projects  and  segments 
account  for  $619.4  million  in  estimated  costs  for  fiscal  year  2000.  Table  4.1 
shows  the  status  of  faa’s  efforts  to  develop  ams  baseline  documentation  on 
the  60  ongoing  projects  and  segments. 


2iSome  of  the  70  baselines  cover  entire  projects,  while  others  cover  segments  of  projects,  meaning  that 
some  projects  have  more  than  one  baseline.  For  example,  4  of  the  70  baselines  are  related  to  the 
Automated  Surface  Observing  System,  2  are  related  to  the  Oceanic  Automation  Program,  2  are  related 
to  the  NAS  Infrastructure  Management  System,  and  2  are  related  to  the  Air  Traffic  Management 
(Infrastructure)  Program. 

22As  noted  in  chapter  2,  parameter  sheets  summarize  a  subset  of  baseline  information  that  is 
designated  for  control  by  the  Joint  Resources  Council  and  that  is  considered  critical  to  assessing  a 
project's  ability  to  satisfy  mission  need,  to  achieve  needed  operational  capability  and  benefits,  and  to 
meet  the  schedule  requirements  of  interdependent  programs. 
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Table  4.1:  Status  of  FAA’s  Efforts  to 
Develop  AMS  Baseline  Documentation 

Dollars  in  millions 

on  60  Ongoing  Projects  and  Segments 
for  Which  Investment  Decisions  Have 
Been  Made 

FAA’s  efforts  to 
develop 
baseline 
documentation 

Number  of 
projects  and 
segments 

Percentage  of 
total  projects 
and  segments 

Fiscal  year 
2000  estimated 
cost  of 
projects  and 
segments 

Percentage  of 
total  fiscal  year 
2000  estimated 
costs 

Acquisition 
program 
baselines  and 
parameter  sheets 
approved  by  the 

Joint  Resources 

Council 

30 

50 

$619.4 

51 

Parameter  sheets 
undergoing 
review  for 
validation  of 
operations  costs 
and  other  funding 
issues 

4 

7 

292.1 

24 

Acquisition 
program 
baselines  and 
parameter  sheets 
being  planned 

26 

43 

293.9 

24 

Total  -  all 

acquisition 

program 

baselines  and 

parameter 

sheets 

60 

100 

$1,205.4 

99J 

Percentages  do  not  add  to  100  because  of  rounding. 


Source:  GAO's  analysis  of  FAA’s  data. 


Our  review  of  five  projects  found  that  some  of  the  baseline  documentation 
(both  acquisition  program  baselines  and  parameter  sheets)  is  incomplete. 
Of  the  five  projects,  two  had  acquisition  program  baselines  (plus 
parameter  sheets),  and  three  had  parameter  sheets  only.  Only  the 
performance  element  was  fully  documented  on  all  of  the  projects,  while 
the  cost,  benefits,  and  schedule  elements  were  missing  some  of  the 
information  required  by  ams.  For  the  cost  baseline,  for  example,  two  of  the 
five  projects  did  not  estimate  operations  costs,  and  one  of  the  two  projects 
with  acquisition  program  baselines  did  not  provide  information  on  the 
detailed  cost  elements  required  by  ams  guidance.  For  the  benefits  baseline, 
one  of  the  two  projects  with  acquisition  program  baselines  did  not  identify 
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the  measurements  that  would  be  used  to  determine  whether  a  benefit  had 
been  achieved.  For  the  schedule  baseline,  none  of  the  projects  estimated 
the  in-service  decision,  a  milestone  required  by  ams  that  specifies  when  the 
newly  acquired  system  or  equipment  is  certified  ready  for  operational  use. 


FAA  Is  Not  Completely 
Monitoring  Projects’ 
Actual  Performance 
Against  Their  Baselines 


Our  review  and  faa’s  internal  evaluation  of  acquisition  reform23  found  that 
the  i-beam  and  acquisition  review  processes  frequently  provide  incomplete 
reporting  on  projects’  estimated  versus  actual  performance  in  the  areas  of 
cost,  schedule,  benefits,  and  performance.  We  reviewed  the  monthly  status 
reports  on  the  five  projects  and  found  information  was  missing  on  all  of 
the  baseline  elements.  For  example,  none  of  the  monthly  status  reports  on 
the  five  projects  analyzed  operations  costs;  assessed  estimated  versus 
actual  benefits,  even  though  benefits  were  projected  for  fiscal  years  1997 
or  1998  on  four  of  the  five  projects;  or  contained  information  on  the 
performance  requirements  outlined  in  the  acquisition  program  baseline  or 
parameter  sheet  to  confirm  whether  the  original  baseline  requirements 
still  applied.  Our  results  are  consistent  with  those  of  faa’s  internal 
evaluation,  which  found  that  the  cost,  schedule,  and  performance  data  in 
the  monthly  status  reports  were  generally  inconsistent  with  the  estimates 
in  the  acquisition  program  baselines  and  parameter  sheets.  With  regard  to 
faa’s  tracking  of  baseline  information  through  the  acquisition  review 
process,  our  review  of  the  five  projects  showed  that  information  was 
missing  for  all  of  the  baseline  parameters,  as  shown  in  table  4.2. 


^Evaluation  of  Acquisition  Reform — The  First  Two  Years:  April  1996-March  1998,  FAA  Program 
Evaluation  Branch,  Office  of  Systems  Architecture  and  Investment  Analysis  (May  29,  1998). 
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Baseline  element 

Missing  information 

Cost 

Four  projects  did  not  analyze  operations  costs. 

Three  projects  did  not  compare  the  original  (or  revised) 
baseline  cost  estimates  with  actual  results  achieved  to 
date. 

Of  the  two  projects  with  acquisition  program  baselines, 
neither  provided  information  on  the  detailed  cost 
elements  required  by  AMS  guidance. 

Schedule 

Three  projects  did  not  compare  the  original  (or  revised) 
baseline  milestones  with  actual  results  achieved  to  date. 

One  project  did  not  address  two  of  the  Joint  Resources 
Council-controlled  milestones:  contract  award  and 
in-service  decision. 

One  project  only  showed  milestones  for  3  years  of  the 
9-year  acquisition  cycle  identified  in  the  parameter  sheet. 

Benefits 

The  acquisition  review  did  not  address  the  baseline 
benefits  in  the  acquisition  program  baselines  or 
parameter  sheets  for  any  of  the  five  projects,  even 
though,  for  four  of  them,  benefits  were  projected  for  fiscal 
years  1997  or  1998. 

Performance 

Four  projects  did  not  address  all  of  the  performance 
parameters  contained  in  the  acquisition  program  baseline 
or  parameter  sheet,  and  one  of  those  projects  had  no 
information  on  performance  at  all. 

Source:  GAO's  analysis  of  FAA's  data. 

faa  officials  managing  the  five  projects  we  reviewed  cited  several  factors 
to  explain  the  lack  of  complete  baseline  data  on  their  projects.  First, 
project  officials  told  us  that  frequent  budget  reductions — imposed  either 
by  the  Congress  or  by  faa — make  it  very  difficult  to  establish  a  stable 
baseline  and  to  monitor  a  project’s  performance  against  that  baseline. 
Second,  ams  guidance  on  the  acquisition  program  baseline  requirements 
has  evolved  over  time,  and  project  officials  told  us  that  detailed  guidance 
was  not  available  to  projects  that  established  their  baselines  during  the 
first  year  of  ams’  implementation.  Finally,  the  two  mechanisms  used  by  faa 
to  monitor  projects’  status — the  monthly  status  report  and  the  acquisition 
reviews — were  developed  prior  to  ams  and  thus  do  not  always  address  the 
specific  baseline  elements  that  are  supposed  to  be  contained  in  the 
acquisition  program  baselines,  report  the  current  status  of  all  baseline 
estimates,  or  report  the  actual  deviations  from  those  estimates. 
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We  recognize  that  variances  in  funding  levels  exist  and,  in  some  cases, 
have  had  an  impact  on  faa’s  ability  to  manage  its  projects.  However,  we 
and  others  have  found  that  faa’s  problems  with  baseline  parameters  have 
resulted  from  factors  other  than  funding.  Although  ams’  detailed  guidance 
and  reporting  requirements  are  evolving,  establishing  complete  acquisition 
program  baseline  documentation  that  has  been  approved  by  the  Joint 
Resources  Council  helps  to  ensure  that  all  projects  are  being  held 
accountable  to  the  cost,  schedule,  benefits,  and  performance  baselines 
established  when  the  Council  made  the  initial  investment  decision. 
Furthermore,  the  absence  of  complete,  up-to-date  data  on  estimated 
versus  actual  results  means  that  faa  has  little  assurance  that  its  estimates 
of  projects’  costs,  schedule,  benefits,  performance,  or  risks  are  sound  and 
accurate  or  that  projects  will  be  managed  so  that  they  meet  the  agency’s 
expectations.  This,  in  turn,  restricts  the  ability  of  faa’s  senior  management 
to  make  sound  decisions  about  continuing  the  agency’s  investments. 

faa’s  long-standing  problems  with  implementing  projects  that  meet  their 
cost,  schedule,  and  performance  objectives  illustrate  the  need  for  the 
agency  to  better  manage  its  baselines.  As  noted  earlier,  over  the  past  17 
years,  faa’s  modernization  projects  have  experienced  substantial  cost 
overruns,  lengthy  delays,  and  significant  performance  shortfalls,  problems 
that  have  persisted  since  the  implementation  of  ams  in  April  1996.  Two  of 
the  projects  in  our  review  provide  examples  of  continuing  cost  and 
schedule  problems.  In  the  case  of  the  Standard  Terminal  Automation 
Replacement  System,  although  faa  has  not  officially  changed  the  baseline 
approved  in  February  1996,  the  baseline  is  in  jeopardy  of  being  breached 
because  of  unions’  concerns  about  human-factor  and  design  issues,24  the 
refinement  of  the  project’s  requirements,  and  the  interjection  of  a  new 
project  phase.26  faa  estimates  that  these  issues  have  the  potential  to 
increase  the  project’s  costs  from  $294  million  to  $410  million  above  the 
approved  baseline,  faa  also  estimates  that  the  project’s  initial  completion 
date  could  be  delayed  by  almost  2-1/2  years.  Similarly,  in  the  case  of  the 
nas  Infrastructure  Management  System,  the  agency  has  not  officially 
changed  the  baseline  that  was  approved  in  March  1997.  However,  the 
project’s  leader  expects  significant  baseline  breaches  to  occur,  including  a 
58-percent  increase  in  the  costs  from  $100.8  million  to  $159.5  million,  and 
a  123-percent  increase  in  the  schedule  from  48  to  107  months  for  the 
project’s  total  duration. 


24Concems  were  raised  by  two  unions,  the  National  Air  Traffic  Controllers  Association  and  the 
Professional  Airways  Systems  Specialists. 

25GAO/RCED-99-25,  Dec.  3, 1998. 
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Joint  Resources  Council 
Has  Limited  Involvement 
in  Reviewing  Projects  After 
the  Investment  Decisions 


Under  ams,  the  Joint  Resources  Council  conducts  reviews  and  makes 
decisions  on  each  project  funded  from  the  facilities  and  equipment 
account,  from  the  determination  of  mission  needs  to  the  point  at  which  an 
investment  decision  is  made.  After  the  investment  decision,  the  Joint 
Resources  Council  generally  only  becomes  directly  involved  in  monitoring 
a  project  when  there  is  a  potential  or  actual  breach  of  the  established 
baseline.  As  a  result,  the  Joint  Resources  Council  is  not  proactively 
involved  in  making  decisions  about  the  future  of  most  projects  when  they 
are  being  developed,  deployed,  operated,  and  maintained;  when  decisions 
are  being  made  about  technology  upgrades  or  other  enhancements;  and 
when  assessments  are  being  made  about  whether  the  system  or  product  is 
achieving  the  intended  benefits  and  meeting  the  expected  performance 
requirements. 


The  Acquisition  Executive,  who  chairs  the  Joint  Resources  Council,  told 
us  that  he  highlights  projects  whose  baseline  status  is  problematic 
according  to  data  in  the  monthly  status  report  and  that  he  provides 
additional  oversight  for  those  projects.  However,  as  we  have  stated,  the 
data  contained  in  the  monthly  status  reports  are  incomplete  and  generally 
inconsistent  with  the  estimated  baseline  elements  in  the  acquisition 
program  baselines  and  parameter  sheets. 


FAA’s  AMS  Lacks  a 
Post-Implementation 
Review  Process  for 
Evaluating 
Investments 


faa  does  not  have  a  defined,  documented  process  for  conducting 
post-implementation  reviews  of  projects  to  assess  their  performance  and 
to  improve  the  selection  and  control  of  its  other  investments,  faa  performs 
some  elements  of  a  post-implementation  review  in  its  life-cycle 
management  review  process,  including  such  tasks  as  the  independent 
operational  test  and  evaluation  of  some  projects  prior  to  their  deployment, 
operational  performance  monitoring,  customer  satisfaction  surveys,  and 
periodic  reviews  throughout  an  investment’s  life  cycle.  However,  this 
process  is  not  standardized  and  is  not  required  for  all  projects.  As  a  result, 
there  is  no  evidence  that  changes,  especially  to  the  selection  and  control 
phases,  are  being  implemented  based  on  lessons  learned.  Although  faa  has 
not  yet  designed  or  implemented  a  post-implementation  review  process 
for  individual  projects,  its  fiscal  year  1999  performance  plan  for  the 
Research  and  Acquisitions  operating  division  includes  a  new  requirement 
for  a  post-deployment  assessment  of  nas  modernization  systems. 


The  evaluation  phase  “closes  the  loop”  on  the  investment  management 
process  by  comparing  actual  results  against  baseline  estimates  to  assess 
performance  and  identify  areas  where  future  decision-making  can  be 
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improved.  Lessons  that  are  learned  during  the  evaluation  phase  can  be 
used  to  improve  future  decisions  about  selecting  and  controlling  projects. 
Central  to  this  process  is  the  post-implementation  review,  with  its 
evaluation  of  the  historical  record  of  a  project  and  its  comparison  of 
actual  versus  expected  costs  and  benefits.  Recognizing  the  importance  of 
this  phase  of  investment  management,  the  Clinger-Cohen  Act  requires 
federal  agencies  to  evaluate  the  performance  of  information  technology 
projects  and  to  use  that  information  to  decide  whether  to  continue, 
modify,  or  terminate  projects. 

At  leading  organizations,  this  review  generally  occurs  about  3  to  12 
months  after  a  project  has  reached  its  end  point  (i.e.,  the  point  at  which 
the  project  has  been  fully  implemented  or  canceled)  and  is  generally 
conducted  by  a  group  other  than  the  project  team  to  ensure  that  the 
review  is  independent  and  objective.  In  conducting  post-implementation 
reviews,  an  organization  can  survey  customers  to  determine  users’ 
satisfaction  with  the  completed  product  and  how  well  the  project  supports 
business  processes;  assess  whether  the  investment  has  had  its  intended 
impacts  on  mission  goals,  cost  savings,  compliance  with  the  system’s 
architecture,  and  other  issues  involving  information  accuracy,  timeliness, 
adequacy,  and  appropriateness;  and  evaluate  current  and  future  technical 
issues  associated  with  the  investment. 


FAA’s  Efforts  to 
Implement  an 
Agencywide 
Management 
Information  System 
Do  Not  Include  Key 
Selection  Data 


Until  recently,  faa  lacked  a  centralized,  standardized  management 
information  system  or  historical  database  for  capturing  and  maintaining 
project  information.  While  faa  had  a  number  of  stand-alone  databases 
within  different  groups,  none  provided  a  complete  picture  of  estimates, 
the  assumptions  that  made  up  the  estimates,  revisions,  and  actual 
performance  on  projects.  As  a  result,  agency  officials  had  no  assurance 
that  the  project  data  from  these  stand-alone  systems  were  complete, 
accurate,  and  up-to-date. 

In  March  1999,  faa  officials  began  implementing  a  centralized  repository 
and  management  information  system  under  ams — called  the  Simplified 
Program  Information  Reporting  and  Evaluation  (spire) — to  provide  access 
to  projects’  baselines  and  other  information.  This  system  consolidates 
project  information  related  to  faa’s  processes  for  managing  key  baselines, 
including  data  from  the  acquisition  program  baselines  and  parameter 
sheets,  monthly  status  reports,  baseline  management  notices,  meeting 
minutes  from  the  acquisition  reviews,  Joint  Resources  Council  decisions, 
and  other  baseline  information.  The  system  does  not,  however,  contain 
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key  information  from  the  selection  process,  such  as  mission  need 
statements,  cost-benefit  analyses,  risk  assessments,  or  other  required 
reports,  faa  plans  to  implement  spire  in  three  phases.  Phase  I,  which 
commenced  in  March  1999,  provides  the  capability  to  store  and  display  the 
status  and  variance  reports  that  have  been  input  by  project  leaders. 
Subsequent  phases,  which  do  not  yet  have  implementation  dates,  will 
focus  on  automatically  generating  various  reports  on  projects’  baseline 
status  and  variances. 

Informed  management  decisions  can  be  made  only  if  accurate,  reliable, 
and  up-to-date  information  from  all  phases  of  the  investment  management 
process  is  included  in  the  decision-making  process.  To  do  this  requires 
that  agencies  have  a  uniform  mechanism — that  is,  a  management 
information  system  with  uniform  data  standards  and  entry 
procedures — for  collecting,  automating,  and  processing  data  on  projects’ 
expected  versus  actual  outcomes.  Data  in  this  system  should  include  the 
initial  cost,  schedule,  benefits,  performance,  and  risk  estimates  that  were 
developed  during  the  selection  process.  Various  analyses  that  were 
conducted  to  initially  justify  the  project,  along  with  revised  estimates, 
reasons  for  revisions,  and  actual  performance  measured  against  the 
estimates,  should  also  be  included.  These  data  need  to  be  continually 
updated  as  each  project’s  implementation  continues  and  as  expenditures 
increase.  The  data  also  need  to  be  easily  accessible  to  both  the  project 
team  and  senior  managers. 

A  management  information  system,  if  kept  accurate  and  up-to-date,  can 
make  data  verification  and  validation  easier  by  allowing  an  organization  to 
track  costs,  risks,  and  other  factors  over  time.  It  is  also  essential  from  the 
standpoint  of  establishing  an  organizational  memory  throughout  the 
selection,  control,  and  evaluation  phases  of  the  investment  management 
process.  As  such,  it  can  be  used  to  help  assess  whether  projects  are  still 
aligned  with  mission  needs  and  organizational  objectives,  determine 
whether  projects  are  meeting  planned  performance  goals,  and  identify 
possible  revisions  to  the  overall  investment  management  process  based  on 
previous  experiences  and  lessons  learned. 

faa’s  efforts  to  consolidate  project  baseline  information  in  spire  are  a 
positive  step  toward  improving  the  agency’s  mechanisms  for  tracking 
projects  and  helping  to  ensure  the  consistency  of  information  and 
reporting  on  all  the  projects  in  its  investment  portfolio.  However,  spire 
provides  an  incomplete  record  of  project  information  since  it  does  not 
include  key  information  from  the  selection  process. 
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Conclusions 


ams  has  weaknesses  in  its  selection,  control,  and  evaluation  phases  that 
impede  faa’s  ability  to  manage  its  investments  effectively  and  to  make 
sound  decisions  about  continuing,  modifying,  or  canceling  projects.  First, 
using  data  that  have  not  been  validated  and  unreliable  cost  information 
reduces  the  likelihood  that  faa’s  managers  can  make  informed  decisions 
about  the  relative  merits  of  competing  investments.  Second,  requiring 
faa’s  senior  managers  to  stay  actively  involved  in  the  process  of 
controlling  project  baselines  and  providing  them  with  complete 
information  on  all  projects  is  critical  to  monitoring  how  well  projects  are 
achieving  their  intended  results.  Third,  establishing  post-implementation 
reviews  is  essential  for  evaluating  projects’  performance  and  identifying 
areas  for  which  future  decision-making  could  be  improved  during  the 
selection  and  control  phases.  Finally,  establishing  a  standardized 
management  information  system  that  includes  complete  information  from 
all  three  investment  management  phases — selection,  control,  and 
evaluation — will  help  facilitate  faa’s  efforts  to  track  projects’  costs,  risks, 
and  other  factors  over  time,  providing  senior  managers  with  uniform, 
accurate,  reliable,  and  easily  accessible  data  on  all  the  projects  in  the 
agency’s  portfolio.  Taking  steps  to  correct  these  weaknesses  increases  the 
likelihood  that  faa’s  projects  will  meet  established  cost  and  schedule 
objectives  and  contribute  to  measurable  improvements  in  the  agency’s 
mission  performance. 


Recommendations 


We  recommend  that  the  Secretary  of  Transportation  direct  the 
Administrator  of  faa  to  take  the  following  actions: 

•  Improve  the  selection  process  by  (1)  establishing  clearly  defined 
procedures  for  validating  projects’  cost,  schedule,  benefit,  performance, 
and  risk  information  and  (2)  requiring  documentation  of  the  results  of  the 
validation  procedures  applied  to  each  project. 

•  Strengthen  control  over  investments  by  (1)  revising  the  acquisition 
program  baseline  requirements  to  include  project  risks  and  to  add 
milestones  for  project  reviews  during  the  operations  phase  and 

(2)  ensuring  that  project  officials  fully  track  and  document  estimated 
versus  actual  results  on  all  the  elements  (i.e.,  costs,  schedule,  benefits, 
performance,  and  risks)  contained  in  the  baseline  documentation. 

•  Initiate  post-implementation  evaluations  for  projects  within  3  to  12  months 
of  deployment  or  cancellation  to  compare  the  completed  projects’  costs, 
schedule,  performance,  and  mission  improvement  outcomes  with  the 
original  estimates. 
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•  Incorporate  key  information  from  the  selection  process  (e.g.,  mission  need 
statements,  cost-benefit  analyses,  and  risk  assessments)  into  faa’s 
management  information  system  for  investments. 


Agency  Comments 


faa  agreed  with  our  recommendations. 
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This  appendix  provides  detailed  information  on  the  purpose,  scope,  and 
status  of  the  five  information  technology  projects  we  reviewed.  The 
Federal  Aviation  Administration  (faa)  considers  each  of  these  projects  key 
to  replacing  the  aging  National  Airspace  System’s  (nas)  infrastructure  or  to 
improving  its  capacity  and  effectiveness.  Each  project  is  estimated  to 
require  the  expenditure  of  more  than  $100  million  from  the  agency’s 
facilities  and  equipment  budget  account  before  it  becomes  operational. 


NAS  Infrastructure 
Management  System 


Background  The  NAS  Infrastructure  Management  System  will  provide  the  next 

generation  of  tools,  services,  and  operational  philosophies  that  govern  the 
management,  operation,  and  maintenance  of  the  nas  infrastructure. 
Currently,  the  air  traffic  control  system  relies  on  a  nationwide 
infrastructure  of  facilities  and  equipment  to  provide  communications, 
navigation,  surveillance,  and  automation  system  capabilities.  To  offset  the 
continual  loss  of  experienced  technical  staff  devoted  to  the  maintenance 
of  the  nas  systems,  faa  has  dispersed  its  maintenance  workforce  into  work 
centers  that,  in  some  cases,  are  colocated  with  “high-impact”  facilities 
such  as  terminal  radar  approach  control  facilities  and  en  route  centers. 

The  agency  is  also  using  a  remote  maintenance  monitoring  system  to 
recognize  and  address  problems  in  a  more  timely  manner.  However,  this 
monitoring  system  simply  collects  performance  data  from  individual 
systems  and  logs  historical  maintenance  actions.  The  nas  Infrastructure 
Management  System  will  build  on  the  investment  made  in  this  monitoring 
system  and  will  focus  on  workforce  and  event  management — two  separate 
but  related  components  of  system  management.  Through  this  project,  faa 
will  be  able  to  track  and  monitor  the  actual  costs  of  providing  nas  services. 

The  mission  need  statement  for  this  project  was  submitted  in  December 
1995  and  revalidated  in  February  1996,  and  a  cost-benefit  analysis  was 
completed  in  December  1995.  faa  approved  the  investment  decision  and 
acquisition  program  baseline  for  Phase  I  of  the  project  in  March  1997. 

According  to  project  officials,  the  nas  Infrastructure  Management  System 
will  be  implemented  in  three  phases.  Only  Phase  I,  which  covers  fiscal 
years  1998  through  2002,  is  fully  defined.  In  this  phase,  faa  plans  to  create 
the  initial  infrastructure  needed  to  manage,  operate,  and  maintain  overall 
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nas  operations.  This  infrastructure  will  consist  of  a  National  Operations 
Control  Center  in  Herndon,  Va.,  and  three  regional  operational  control 
centers  in  Hampton,  Ga.;  Olathe,  Kans.;  and  San  Diego,  Calif.  Below  these 
three  regional  centers  will  be  up  to  50  service  operations  centers  modeled 
after  the  facilities  that  are  currently  colocated  with  high-impact  facilities. 
Under  Phase  II,  faa  plans  to  provide  centralized  asset  management, 
develop  customer  and  user  interaction  tools,  and  analyze  technical  and 
cost  trends.  Under  Phase  HI,  the  agency  plans  to  provide  intelligent  fault 
correlation,  information  sharing,  and  additional  functionality 
commensurate  with  nas  technological  improvements. 


Status  According  to  faa,  the  National  Operations  Control  Center  became 

operational  in  January  1999.  The  three  regional  centers  are  scheduled  to 
begin  operations  in  October  2000.  Phase  I  of  the  nas  Infrastructure 
Management  System  will  support  faa’s  Free  Flight  Phase  I  in  fiscal  years 
2000-2002.26  Phases  II  and  III  are  currently  undergoing  an  investment 
analysis  to  determine  if  further  investment  is  warranted. 

Although  faa  has  not  officially  changed  the  project’s  baseline  that  was 
approved  in  1997,  the  project’s  leader  estimated  significant  breaches  in 
December  1998,  as  follows:  (1)  the  cost  baseline  is  expected  to  increase  by 
58  percent,  from  $100.8  million  to  $159.5  million,  and  (2)  the  schedule  is 
expected  to  increase  by  123  percent,  from  48  to  107  months’  total  duration 
for  Phase  I. 


Oceanic  Automation 
Program 


Background  The  Oceanic  Automation  Program  is  designed  to  provide  a  platform  for 

improved  air  traffic  control  over  the  oceans.  It  evolved  as  a  series  of 
projects — each  advancing  on  the  technology  of  its  predecessor — from  the 
Oceanic  Display  and  Planning  System  into  the  Oceanic  Automation 
System,  and  now,  into  the  Advanced  Oceanic  Automation  System.  In  the 
late  1980s,  the  Oceanic  Display  and  Planning  System  improved  oceanic 
traffic  control  by  providing  flight  data  processing  and  a  situational  display 
of  estimated  aircraft  positions.  This  system  also  alerted  controllers  when 


2fiFree  flight  is  a  new  system  of  air  traffic  management  that  will  provide  controllers  and  pilots  with  new 
technologies  and  procedures  that  will  allow  them  to  increase  the  safety,  capacity,  and  efficiency  of  air 
traffic  operations  throughout  the  NAS. 
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any  flight  plan  or  any  route  change  requested  by  a  pilot  (referred  to  as 
conflict  probe  capability)  violated  appropriate  separation  standards.  In  the 
early  1990s,  faa  introduced  the  Oceanic  Automation  System,  which 
improved  data  display  and  communications.  This  system  is  now  being 
upgraded  to  the  Advanced  Oceanic  Automation  System,  which  is  designed 
to  provide  such  features  as  a  new  flight  data  processor,  automatic 
dependent  surveillance  position  reporting,  an  advanced  conflict  probe, 
and  data  link,  faa  awarded  a  contract  to  the  Raytheon  Systems 
Corporation  in  September  1995  for  the  Advanced  Oceanic  Automation 
System.  The  contract  is  composed  of  flexible  segments,  which  will  allow 
for  incremental  functional  development  and  delivery  of  benefits.  Oceanic 
air  traffic  control  systems  are  installed  at  the  en  route  centers  at  Oakland 
and  New  York  and  in  Anchorage,  Alaska, 

The  mission  need  statement  was  approved  in  May  1992.  In  October  1992, 
the  acquisition  plan  was  approved  to  consolidate  and  integrate  the  primary 
oceanic  improvement  projects  into  a  single  Oceanic  Automation  Program. 
A  revised  mission  need  statement  and  acquisition  plan  was  approved  in 
January  1994. 


Status  Since  faa  awarded  the  Advanced  Oceanic  Automation  System’s  contract  in 

September  1995,  the  scope  of  the  project  has  been  gradually  cut  back  from 
an  original  plan  of  five  segments  (that  is,  five  incremental  deliveries  of 
capabilities)  to  only  a  portion  of  the  first  segment.  In  July  1996, 10  months 
after  the  contract’s  award,  faa  canceled  segments  3,  4,  and  5  of  the  project 
because  the  agency  recognized  that  the  cost  of  executing  these  segments 
was  beyond  the  funding  that  had  been  allocated  for  this  project.  As  a 
result,  faa  abandoned  many  controller  productivity  tools  needed  to 
increase  the  system’s  capacity.  Then,  in  December  1996,  funding  concerns 
forced  faa  to  revise  the  Segment  2  of  the  project,  which  was  designed  to 
replace  infrastructure  hardware  and  software  that  supports  controller 
equipment.  Eventually,  in  September  1997,  faa  canceled  the  entire 
Segment  2  because  the  agency  needed  to  use  the  project’s  funds  to  correct 
Year  2000  problems  in  existing  oceanic  automation  software  and  because 
it  needed  to  transfer  funds  to  the  Host  replacement  program.27 

Meanwhile,  faa’s  contractor  was  reporting  performance  problems  with  the 
Segment  1  of  the  project,  which  adds  data  link  and  automatic  dependent 
surveillance  in  the  oceanic  environment.  To  avoid  a  potential  $45  million 


27The  Host  replacement  project  replaces  en  route  center  and  oceanic  automation  hardware  that  has 
reached  the  end  of  its  commercial  support  life  and  may  have  problems  with  Year  2000  date 
requirements. 
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cost  overrun  for  this  segment,  faa  reduced  its  scope  in  September  1998  by 
eliminating  the  capability  for  automatic  dependent  surveillance.  According 
to  project  officials,  the  remaining  elements  of  Segment  1  (the  air-to-ground 
data  link,  the  ground-to-ground  data  link,  and  controller  tools)  have 
successfully  completed  the  operational  test  and  evaluation  and  are 
expected  to  be  delivered  on  schedule.  The  last  site  implementation  is 
estimated  for  October  1999. 


Operational  and 
Supportability 
Implementation 
System 


Background  The  Operational  and  Supportability  Implementation  System  project 

(1)  replaces  the  Flight  Service  Automation  System’s  hardware  and 
software  with  a  leased  commercial,  off-the-shelf-based  service; 

(2)  provides  an  improved  graphic  weather  display  capability;  and 

(3)  incorporates  direct  user  access  functionality  that  is  currently  being 
obtained  through  two  direct  user  access  terminal  contracts.  The 
integration  of  these  three  capabilities  and  functions  into  a  single  system 
will  enable  flight  service  specialists  to  more  efficiently  provide  weather 
and  flight-planning  information  for  pilots. 

The  mission  need  statement  was  approved  in  October  1993  and  revalidated 
in  December  1996.  The  acquisition  program  baseline  was  approved  in 
April  1997,  and  in  August  1997,  faa  awarded  a  contract  to  Harris 
Corporation  for  the  project.  The  contract  requires  Harris  to  provide  up  to 
61  operational  systems  and  3  support  systems. 


Status  Since  faa  awarded  the  contract,  the  project’s  schedule  has  slipped  because 

the  development  effort  has  been  larger  than  planned,  faa’s  January  1998 
review  of  the  Harris  system’s  architecture  for  the  project  revealed  that  the 
contractor’s  commercial,  off-the-shelf  solution  was  not  as  mature  as  faa 
had  envisioned  when  the  contract  was  awarded  and  that  many  of  the 
contractor’s  commercial  products  did  not  fully  satisfy  faa’s  requirements. 
In  May  1998,  the  agency  decided  to  replace  workstation  consoles  in 
response  to  human-factor  concerns  raised  by  the  unions  that  represent  its 
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controllers  and  the  technicians.  This  caused  the  project’s  costs  to  increase 
by  $15.8  million,  faa  also  delayed  first-site  implementation  from  July  1998 
to  January  1999 — a  6-month  slip.  The  protracted  development  effort  is  not 
expected  to  delay  the  completion  of  the  project,  with  the  last  site  to 
receive  the  system  still  scheduled  to  be  operational  in  August  2001. 


Standard  Terminal 
Automation 
Replacement  System 


Background  The  Standard  Terminal  Automation  Replacement  System  is  designed  to 

replace  faa’s  automated  radar  terminal  system,  which  comprises  15-  to 
25-year-old  air  traffic  controller  workstations  and  the  supporting  computer 
systems  that  allow  controllers  at  terminal  radar  approach  control  facilities 
to  separate  and  sequence  aircraft.  According  to  faa,  the  old  system  is 
prone  to  failures  and  requires  extensive  maintenance.  The  old  system  also 
has  capacity  constraints  that  restrict  the  agency  from  making  required 
safety  and  efficiency  enhancements.  Besides  remedying  those  problems, 
the  new  equipment  is  also  expected  to  allow  the  system  to  increase  the 
level  of  air  traffic  control  automation  and  to  improve  surveillance, 
communications,  and  weather  display.  This  system  replaced  a  segment  of 
another  project  (the  Advanced  Automation  System)  that  was  terminated 
because  of  serious  cost  and  schedule  problems. 

The  mission  need  statement  for  the  Standard  Terminal  Automation 
Replacement  System  was  submitted  in  July  1993  and  revised  in  June  1995. 
The  acquisition  plan  was  approved  in  March  1996,  and  in  September  1996, 
faa  signed  a  contract  with  Raytheon  Corporation  to  acquire  this  system.  In 
producing  it,  Raytheon  originally  intended  to  rely  exclusively  on 
commercially  available  hardware  and,  to  a  large  extent,  on  commercially 
available  software. 

The  initial  strategy  for  replacing  and  enhancing  the  system  is  divided  into 
two  stages.  Stage  1  is  expected  to  provide  the  same  functions  as  the 
current  automated  radar  terminal  systems.  Stage  2  is  expected  to 
implement  new  functions  to  help  controllers  move  aircraft  more  safely  and 
efficiently.  In  1997,  faa  created  another  stage,  known  as  early  display 
configuration,  because  of  concerns  about  operational  problems  at  Ronald 
Reagan  National  Airport  in  Washington,  D.C.  This  new  stage  will  be 
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implemented  prior  to  Stages  1  and  2.  The  new  stage  replaces  the  current 
controller  displays  and  monitoring  equipment  but  uses  the  existing 
computer  system  and  software.  It  also  provides  an  emergency  backup 
system. 


Status  Although  faa  has  not  officially  changed  the  project’s  baseline  that  was 

approved  in  1996,  the  baseline  is  in  jeopardy  of  being  breached  because  of 
the  unions’  concerns  about  human-factor  and  design  issues,  the 
refinement  of  the  requirements,  and  the  interjection  of  a  new  project 
phase,  faa  estimates  that  these  issues  have  the  potential  to  increase  the 
project’s  costs  anywhere  from  $294  million  to  $410  million  over  the 
approved  baseline,  faa  also  estimates  that  the  project’s  initial  completion 
date  could  be  delayed  by  almost  2-1/2  years.  In  addition,  the  project  has 
experienced  other  challenges  mainly  involving  software  testing.  While 
project  officials  stated  that  they  have  been  able  to  absorb  the  cost 
increases  associated  with  this  issue  within  the  existing  baseline,  additional 
problems  could  cause  further  cost  increases  and  schedule  delays.  The  last 
site  implementation  is  estimated  for  February  2005. 


Voice  Switching  and 
Control  System 


Background  The  Voice  Switching  and  Control  System  replaces  existing  communication 

systems  at  en  route  centers  with  an  expandable,  highly  reliable  system  for 
both  ground-to-ground  and  air-to-ground  communication.  This  system  will 
also  provide  communication  capability  for  new  en  route  center  controller 
workstations  that  are  being  installed,  faa  is  also  installing  the  Voice 
Switching  and  Communications  System  Training  and  Backup  Switch — an 
emergency  backup  communications  system — at  all  en  route  centers. 

This  system  was  designed  to  provide  the  communication  capabilities  for 
the  new  Initial  Sector  Suite  System  workstations  under  the  Advanced 
Automation  System  program.  By  the  time  the  contract  was  awarded  in 
December  1991  to  the  Harris  Corporation,  faa  had  spent  5  years 
developing  prototypes  and  had  incurred  cost  growth  of  about  $1  billion.28 
The  contract  required  Harris  to  deliver  23  systems — 21  for  en  route 


28According  to  project  officials,  the  primary  reason  for  this  growth  was  the  inability  of  commercially 
available  products  to  effectively  and  accurately  manage  air  traffic  control  communications  functions. 
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centers  and  2  support  systems,  faa’s  plans  called  for  the  Voice  Switching 
and  Communications  System  to  be  installed  with  both  the  current 
equipment  and  with  the  new  controller  workstations.  During  the  initial 
development,  the  cost  of  the  project  increased  by  $53.1  million  to 
approximately  $1.45  billion — primarily  because  of  faa’s  decision  in  1994  to 
cancel  the  Initial  Sector  Suite  System  component  of  the  Advanced 
Automation  System  and  replace  it  with  the  Display  System  Replacement 
project.  The  restructuring  resulted  in  the  need  for  additional  equipment 
and  testing  and  in  the  retention  of  contractor  and  project  personnel  longer 
than  planned  to  field  the  communications  equipment  with  new  controller 
workstation  equipment,  faa  has  also  added  new  functionality  requirements 
to  the  project. 

The  original  concept  for  this  project  was  defined  in  1980.  In  1984, 
operational  requirements  were  finalized,  and  in  1985,  the  project  was 
approved  for  development.  A  revised  draft  mission  need  statement  was 
completed  in  January  1994. 


Status 


Harris  developed  and  installed  the  system  in  the  existing  en  route 
controller  work  stations  in  February  1997 — 5  months  ahead  of  the 
schedule  established  at  the  time  the  contract  was  awarded.  Harris  is 
currently  reinstalling  the  controller  interface  equipment  into  the  en  route 
Display  System  Replacement  controller  workstations.  Harris  has 
completed  the  software  development  for  the  primary  system  to  be  fielded 
with  the  new  Display  System  Replacement  controller  workstations. 
According  to  the  project  manager,  the  project  has  not  encountered  any 
technical  problems  and  is  not  expected  to  incur  any  major  delays.  The  last 
site  implementation  is  estimated  for  May  2000. 

faa  is  in  the  process  of  installing  the  emergency  backup  system  at  en  route 
centers,  and  by  November  1999,  faa  expects  to  have  completed 
installation  at  all  en  route  centers,  the  faa  Technical  Center,  and  the  faa 
Academy. 
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